[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:30 dist-test-slave-m8jb krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:30 dist-test-slave-m8jb krb5kdc[399](info): set up 2 sockets
Sep 24 19:31:30 dist-test-slave-m8jb krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 19:31:31.092770 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:31.204041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35155
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:31.281432 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:31.281632 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:31.281661 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:31.282810 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:31.282852 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:31.282868 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:31.282886 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:31.282899 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:31.284345 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:31.284590 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:31.284843 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:31.289958 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:31.289945 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:31.290064 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:31.290120 412 server_base.cc:1034] running on GCE node
I20260924 19:31:31.290335 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:31.290619 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:31.291791 412 hybrid_clock.cc:648] HybridClock initialized: now 1790278291291775 us; error 36 us; skew 500 ppm
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:31.294203 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:31.294642 412 webserver.cc:460] Webserver started at http://127.0.98.62:42093/ using document root <none> and password file <none>
I20260924 19:31:31.294847 412 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:31.294889 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:31.295008 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:31.296056 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "8d0827d485b040da80ed504069e47bc1"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "5a8c767ab496ebd8a05cbefaae35298d"
server_key_iv: "023fbf94220ed09d7f4692104d4b08ae"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.296378 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d0827d485b040da80ed504069e47bc1"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "5a8c767ab496ebd8a05cbefaae35298d"
server_key_iv: "023fbf94220ed09d7f4692104d4b08ae"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.297645 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:31.298321 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.298457 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:31.298542 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "8d0827d485b040da80ed504069e47bc1"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "5a8c767ab496ebd8a05cbefaae35298d"
server_key_iv: "023fbf94220ed09d7f4692104d4b08ae"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.298609 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:31.317528 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:31.319506 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:31.319638 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:31.323361 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35155
I20260924 19:31:31.323405 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35155 every 8 connection(s)
I20260924 19:31:31.323733 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:31.324697 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:31.326982 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1: Bootstrap starting.
I20260924 19:31:31.327582 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:31.327843 480 log.cc:824] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:31.328296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 412
I20260924 19:31:31.328382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:31.328517 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1: No bootstrap required, opened a new log
I20260924 19:31:31.329038 392 external_mini_cluster.cc:1428] Setting key 70a65c509ebcc1f28a7694d0841f03a7
I20260924 19:31:31.330418 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0827d485b040da80ed504069e47bc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35155 } }
I20260924 19:31:31.330559 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:31.330591 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0827d485b040da80ed504069e47bc1, State: Initialized, Role: FOLLOWER
I20260924 19:31:31.330722 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0827d485b040da80ed504069e47bc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35155 } }
I20260924 19:31:31.330794 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:31.330824 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:31.330870 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:31.331676 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0827d485b040da80ed504069e47bc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35155 } }
I20260924 19:31:31.331768 480 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0827d485b040da80ed504069e47bc1; no voters:
I20260924 19:31:31.331926 480 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:31.331970 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:31.332149 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [term 1 LEADER]: Becoming Leader. State: Replica: 8d0827d485b040da80ed504069e47bc1, State: Running, Role: LEADER
I20260924 19:31:31.332162 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:31.332284 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0827d485b040da80ed504069e47bc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35155 } }
I20260924 19:31:31.332686 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0827d485b040da80ed504069e47bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0827d485b040da80ed504069e47bc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35155 } } }
I20260924 19:31:31.332757 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:31.332935 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:31.333520 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:31.334204 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0827d485b040da80ed504069e47bc1. Latest consensus state: current_term: 1 leader_uuid: "8d0827d485b040da80ed504069e47bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0827d485b040da80ed504069e47bc1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35155 } } }
I20260924 19:31:31.334419 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:31.335134 490 catalog_manager.cc:1347] Generated new cluster ID: 3f9cb9b2c4b24f22b52739de0de76898
I20260924 19:31:31.335178 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:31.345986 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.329907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.0.98.62:35155)
0924 19:31:31.330098 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:31.330108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:31.330336 (+ 228us) server_negotiation.cc:407] Connection header received
0924 19:31:31.332151 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:31.332159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:31.332215 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:31.332359 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:31.334423 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.335767 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.337226 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.337872 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.337988 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.342082 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:31.342086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:31.342086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:31.342099 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:31.343989 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.344481 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.344484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.344484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.344605 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.344883 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.344884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.344884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.345176 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:31.345310 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:31.345614 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:31:31.345746 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260924 19:31:31.350771 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:31.351388 490 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:31.354418 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0827d485b040da80ed504069e47bc1: Generated new TSK 0
I20260924 19:31:31.354616 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:31.387429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35155
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:31.465438 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:31.465646 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:31.465675 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:31.466779 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:31.466817 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:31.466852 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:31.468254 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:31.468474 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:31.468701 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:31:31.473647 506 server_base.cc:1034] running on GCE node
W20260924 19:31:31.473762 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:31.473627 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:31.473644 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:31.474176 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:31.474392 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:31.475524 506 hybrid_clock.cc:648] HybridClock initialized: now 1790278291475510 us; error 31 us; skew 500 ppm
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:31.477887 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:31.478307 506 webserver.cc:460] Webserver started at http://127.0.98.1:41207/ using document root <none> and password file <none>
I20260924 19:31:31.478495 506 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:31.478538 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:31.478654 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:31.479679 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "6261ea9c4c1e42b7ac40dcafd2e3d8bc"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "51ffe18cdc15b845b9cd7b9d45342e06"
server_key_iv: "39a5e550310aa8928d5722927c5406e0"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.480008 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6261ea9c4c1e42b7ac40dcafd2e3d8bc"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "51ffe18cdc15b845b9cd7b9d45342e06"
server_key_iv: "39a5e550310aa8928d5722927c5406e0"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.481488 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:31:31.482215 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.482353 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:31.482463 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "6261ea9c4c1e42b7ac40dcafd2e3d8bc"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "51ffe18cdc15b845b9cd7b9d45342e06"
server_key_iv: "39a5e550310aa8928d5722927c5406e0"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.482529 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:31.490155 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:31.492003 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:31.492112 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:31.492311 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:31.492599 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:31.492659 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.492689 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:31.492702 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.498864 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45379
I20260924 19:31:31.498903 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45379 every 8 connection(s)
I20260924 19:31:31.499210 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:31:31.501961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 506
I20260924 19:31:31.502084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:31.502247 392 external_mini_cluster.cc:1428] Setting key 7bd5cba6f63f926f93e751b76f1e042c
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:31.509106 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.499895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40141 (local address 127.0.98.62:35155)
0924 19:31:31.499951 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:31.499953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:31.500147 (+ 194us) server_negotiation.cc:407] Connection header received
0924 19:31:31.500626 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:31.500651 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:31.500666 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:31.500719 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:31.501661 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.502369 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.503768 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.503968 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.504026 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.506365 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:31.506366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:31.506366 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:31.506373 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:31.507681 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.508156 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.508157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.508157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.508203 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.508459 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.508459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.508459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.508564 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:31.508607 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:31.508957 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 19:31:31.509039 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:31:31.509421 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.499887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35155 (local address 127.0.98.1:40141)
0924 19:31:31.500030 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:31.500072 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:31.500531 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:31.500792 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:31.500795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:31.501382 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:31.501578 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:31.501583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.502502 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.502503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:31.503685 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:31.503688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.504065 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.504066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:31.504256 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.504845 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:31.504854 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:31.506268 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:31.507763 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:31.507771 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:31.507773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:31.508085 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:31.508256 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:31.508257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:31.508257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:31.508397 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:31.508620 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:31.508625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:31.508821 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 19:31:31.508943 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 19:31:31.509123 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260924 19:31:31.509967 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35155
I20260924 19:31:31.510067 635 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:31.510246 635 heartbeater.cc:511] Master 127.0.98.62:35155 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 19:31:31.510753 444 ts_manager.cc:194] Registered new tserver with Master: 6261ea9c4c1e42b7ac40dcafd2e3d8bc (127.0.98.1:45379)
I20260924 19:31:31.511350 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:31.527241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35155
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:31.605245 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:31.605469 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:31.605502 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:31.606648 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:31.606689 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:31.606724 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:31.608740 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:31.608991 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:31.609212 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:31.615096 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:31.615068 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:31.615384 640 server_base.cc:1034] running on GCE node
W20260924 19:31:31.615487 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:31.615715 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:31.615924 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:31.617056 640 hybrid_clock.cc:648] HybridClock initialized: now 1790278291617042 us; error 29 us; skew 500 ppm
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:31.619453 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:31.619869 640 webserver.cc:460] Webserver started at http://127.0.98.2:40563/ using document root <none> and password file <none>
I20260924 19:31:31.620077 640 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:31.620122 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:31.620240 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:31.621290 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b5b48b679b445a4961bfe8229a4433e"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "68406957d484b6f4de18dd31735e8e7b"
server_key_iv: "00e01bb370df83dc03800c0d78d55c77"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.621614 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b5b48b679b445a4961bfe8229a4433e"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "68406957d484b6f4de18dd31735e8e7b"
server_key_iv: "00e01bb370df83dc03800c0d78d55c77"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.622882 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:31.623584 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.623728 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:31.623807 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "1b5b48b679b445a4961bfe8229a4433e"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "68406957d484b6f4de18dd31735e8e7b"
server_key_iv: "00e01bb370df83dc03800c0d78d55c77"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.623867 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:31.643630 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:31.645730 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:31.645851 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:31.646078 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:31.646365 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:31.646399 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.646420 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:31.646443 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.652246 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46325
I20260924 19:31:31.652308 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46325 every 8 connection(s)
I20260924 19:31:31.652592 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:31.660856 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.653275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50081 (local address 127.0.98.62:35155)
0924 19:31:31.653337 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:31.653339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:31.653517 (+ 178us) server_negotiation.cc:407] Connection header received
0924 19:31:31.654010 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:31.654011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:31.654024 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:31.654069 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:31.654996 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.655483 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.656452 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.656673 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.656732 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.658613 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:31.658615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:31.658615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:31.658621 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:31.659815 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.660132 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.660133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.660133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.660176 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.660354 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.660355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.660355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.660448 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:31.660492 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:31.660732 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:31:31.660793 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:31:31.661037 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.653251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35155 (local address 127.0.98.2:50081)
0924 19:31:31.653404 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:31.653438 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:31.653911 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:31.654142 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:31.654146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:31.654706 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:31.654898 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:31.654903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.655568 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.655569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:31.656365 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:31.656367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.656740 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.656741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:31.656860 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.657265 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:31.657273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:31.658527 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:31.659876 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:31.659879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:31.659881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:31.660073 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:31.660220 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:31.660220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:31.660220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:31.660312 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:31.660488 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:31.660491 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:31.660604 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 19:31:31.660716 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:31:31.660832 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260924 19:31:31.661470 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35155
I20260924 19:31:31.661571 769 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:31.661641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 640
I20260924 19:31:31.661729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:31.661757 769 heartbeater.cc:511] Master 127.0.98.62:35155 requested a full tablet report, sending...
I20260924 19:31:31.661893 392 external_mini_cluster.cc:1428] Setting key 426a437dfeae9cdef432f71b5974a451
I20260924 19:31:31.662156 444 ts_manager.cc:194] Registered new tserver with Master: 1b5b48b679b445a4961bfe8229a4433e (127.0.98.2:46325)
I20260924 19:31:31.662700 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50081
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:31.685243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35155
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:31.763418 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:31.763626 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:31.763656 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:31.764796 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:31.764833 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:31.764868 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:31.766359 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35155
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:31.766593 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:31.766831 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:31.771553 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:31.771688 774 server_base.cc:1034] running on GCE node
W20260924 19:31:31.771612 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:31.772059 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:31.772265 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:31.772464 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:31.773594 774 hybrid_clock.cc:648] HybridClock initialized: now 1790278291773580 us; error 27 us; skew 500 ppm
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:31.775820 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:31.776212 774 webserver.cc:460] Webserver started at http://127.0.98.3:40963/ using document root <none> and password file <none>
I20260924 19:31:31.776401 774 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:31.776448 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:31.776574 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:31.777601 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "4e6cf761037f4f7592ca66a17b40d41b"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "4eb49c79bfc528fb9ff4edd5f6d70719"
server_key_iv: "62676c04901dc5b96f358baf276a704d"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.777922 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4e6cf761037f4f7592ca66a17b40d41b"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "4eb49c79bfc528fb9ff4edd5f6d70719"
server_key_iv: "62676c04901dc5b96f358baf276a704d"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.779160 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:31.779786 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.779930 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:31.780009 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "4e6cf761037f4f7592ca66a17b40d41b"
format_stamp: "Formatted at 2026-09-24 19:31:31 on dist-test-slave-m8jb"
server_key: "4eb49c79bfc528fb9ff4edd5f6d70719"
server_key_iv: "62676c04901dc5b96f358baf276a704d"
server_key_version: "encryptionkey@0"
I20260924 19:31:31.780067 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:31.797465 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:31.799284 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:31.799418 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:31.799629 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:31.799917 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:31.799949 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.799978 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:31.799995 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:31.806041 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44827
I20260924 19:31:31.806102 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44827 every 8 connection(s)
I20260924 19:31:31.806384 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:31:31.809110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 774
I20260924 19:31:31.809195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:31.809347 392 external_mini_cluster.cc:1428] Setting key 649eb65395ef02d1b5dec7ffdcfd2d33
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:31.814769 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.807045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47541 (local address 127.0.98.62:35155)
0924 19:31:31.807094 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:31.807096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:31.807278 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:31:31.807733 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:31.807734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:31.807742 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:31.807799 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:31.808707 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.809224 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.810481 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.810674 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.810725 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.812540 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:31.812542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:31.812542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:31.812548 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:31.813784 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.814101 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.814102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.814102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.814145 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.814313 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.814314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.814314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.814408 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:31.814452 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:31.814653 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 19:31:31.814708 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:31:31.814921 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.807016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35155 (local address 127.0.98.3:47541)
0924 19:31:31.807161 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:31.807203 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:31.807662 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:31.807853 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:31.807856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:31.808389 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:31.808589 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:31.808593 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.809304 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.809305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:31.810406 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:31.810408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.810737 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.810738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:31.810832 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.811204 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:31.811211 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:31.812468 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:31.813849 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:31.813853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:31.813855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:31.814045 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:31.814189 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:31.814190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:31.814190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:31.814271 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:31.814449 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:31.814451 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:31.814561 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 19:31:31.814633 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:31:31.814743 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260924 19:31:31.815311 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35155
I20260924 19:31:31.815402 903 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:31.815562 903 heartbeater.cc:511] Master 127.0.98.62:35155 requested a full tablet report, sending...
I20260924 19:31:31.815889 444 ts_manager.cc:194] Registered new tserver with Master: 4e6cf761037f4f7592ca66a17b40d41b (127.0.98.3:44827)
I20260924 19:31:31.816241 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47541
I20260924 19:31:31.821358 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:31.842906 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.836066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.0.98.1:45379)
0924 19:31:31.836222 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:31.836227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:31.836235 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:31.836258 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:31.836260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:31.836273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:31.836334 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:31.836915 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.837641 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.838354 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.838687 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.838744 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.840793 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:31.840800 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:31.840801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:31.840808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:31.842009 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.842311 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.842313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.842314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.842363 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.842485 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.842486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.842486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.842605 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:31.842715 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:31.842770 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 19:31:31.842829 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
W20260924 19:31:31.843245 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51984
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:31.862787 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:31.856889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.0.98.1:45379)
0924 19:31:31.856956 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:31.856958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:31.857012 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:31:31.857089 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:31.857090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:31.857104 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:31.857155 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:31.857626 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.858125 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.858788 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:31.858976 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:31.859018 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:31.860913 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:31.860914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:31.860915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:31.860921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:31.861999 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.862268 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.862269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.862269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.862311 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:31.862427 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:31.862427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:31.862427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:31.862505 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:31.862598 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:31.862646 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 19:31:31.862706 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:31:31.863261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 506
I20260924 19:31:31.867187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 640
I20260924 19:31:31.871809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 774
I20260924 19:31:31.875633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 412
2026-09-24T19:31:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1759 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[917](info): set up 2 sockets
Sep 24 19:31:31 dist-test-slave-m8jb krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 19:31:32.823967 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:32 dist-test-slave-m8jb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:32.934922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36551
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:33.014107 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:33.014318 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:33.014340 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:33.015483 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:33.015516 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:33.015529 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:33.015540 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:33.015550 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:33.017040 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:33.017243 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:33.017416 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:33.022176 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:33.022166 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:33.022176 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:33.022210 930 server_base.cc:1034] running on GCE node
I20260924 19:31:33.022650 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:33.022940 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:33.024091 930 hybrid_clock.cc:648] HybridClock initialized: now 1790278293024077 us; error 32 us; skew 500 ppm
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:33.026527 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:33.026925 930 webserver.cc:460] Webserver started at http://127.0.98.62:40665/ using document root <none> and password file <none>
I20260924 19:31:33.027105 930 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:33.027148 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:33.027233 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:33.028249 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "e490df220d334b328929c0e73cc5806c"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "06710694d21c5c903325a44576eecb4c"
server_key_iv: "94cdd0fee590983fc69e94776ef69fe9"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.028537 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "e490df220d334b328929c0e73cc5806c"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "06710694d21c5c903325a44576eecb4c"
server_key_iv: "94cdd0fee590983fc69e94776ef69fe9"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.029814 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:33.030546 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.030735 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:33.030814 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "e490df220d334b328929c0e73cc5806c"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "06710694d21c5c903325a44576eecb4c"
server_key_iv: "94cdd0fee590983fc69e94776ef69fe9"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.030867 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:33.052142 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:33.053910 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:33.054028 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:33.057610 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36551
I20260924 19:31:33.057651 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36551 every 8 connection(s)
I20260924 19:31:33.057967 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:33.058801 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:33.059365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 930
I20260924 19:31:33.059437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:33.059571 392 external_mini_cluster.cc:1428] Setting key 2c5b2cbef83676ba190f8e6f5cc4e166
I20260924 19:31:33.061543 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c: Bootstrap starting.
I20260924 19:31:33.062213 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:33.062502 998 log.cc:824] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:33.063141 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c: No bootstrap required, opened a new log
I20260924 19:31:33.065182 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e490df220d334b328929c0e73cc5806c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } }
I20260924 19:31:33.065315 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:33.065335 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e490df220d334b328929c0e73cc5806c, State: Initialized, Role: FOLLOWER
I20260924 19:31:33.065445 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e490df220d334b328929c0e73cc5806c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } }
I20260924 19:31:33.065514 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:33.065541 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:33.065591 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 0 FOLLOWER]: Advancing to term 1
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:33.066073 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e490df220d334b328929c0e73cc5806c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } }
I20260924 19:31:33.066179 998 leader_election.cc:302] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e490df220d334b328929c0e73cc5806c; no voters:
I20260924 19:31:33.066324 998 leader_election.cc:288] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:33.066365 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:33.066516 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [term 1 LEADER]: Becoming Leader. State: Replica: e490df220d334b328929c0e73cc5806c, State: Running, Role: LEADER
I20260924 19:31:33.066568 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:33.066640 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e490df220d334b328929c0e73cc5806c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } }
I20260924 19:31:33.067263 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e490df220d334b328929c0e73cc5806c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e490df220d334b328929c0e73cc5806c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } }
I20260924 19:31:33.067636 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:33.067538 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e490df220d334b328929c0e73cc5806c. Latest consensus state: current_term: 1 leader_uuid: "e490df220d334b328929c0e73cc5806c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e490df220d334b328929c0e73cc5806c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36551 } } }
I20260924 19:31:33.067744 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:33.068820 1020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:33.069293 1020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:33.070007 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.060064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.0.98.62:36551)
0924 19:31:33.060239 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.060249 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.060282 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:31:33.060824 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.060833 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.060906 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.061052 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:33.061581 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.062856 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.063608 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.064214 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.064326 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.066484 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:33.066488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:33.066489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:33.066501 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:33.068504 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.068896 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.068899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.068899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.068997 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.069170 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.069171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.069171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.069435 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:33.069591 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.069682 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.069812 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260924 19:31:33.071020 1020 catalog_manager.cc:1347] Generated new cluster ID: b7fd2f7f149d49cca6ecd8d45411d79f
I20260924 19:31:33.071077 1020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:33.083766 1020 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:33.084528 1020 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:33.092654 1020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e490df220d334b328929c0e73cc5806c: Generated new TSK 0
I20260924 19:31:33.092870 1020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:33.117095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36551
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:33.195199 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:33.195399 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:33.195421 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:33.196521 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:33.196556 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:33.196580 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:33.197991 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:33.198203 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:33.198383 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:33.203131 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:33.203150 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:33.203195 1025 server_base.cc:1034] running on GCE node
W20260924 19:31:33.203261 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:33.203438 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:33.203676 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:33.204810 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790278293204796 us; error 28 us; skew 500 ppm
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:33.207064 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:33.207486 1025 webserver.cc:460] Webserver started at http://127.0.98.1:43825/ using document root <none> and password file <none>
I20260924 19:31:33.207695 1025 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:33.207741 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:33.207860 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:33.208931 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "d0bf1fa871b34d2597b0a71cc3830fdf"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "e34e2b9267cd8f0351eed724ddbd4429"
server_key_iv: "51cbecae63be46cc9b42542823f9595e"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.209244 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d0bf1fa871b34d2597b0a71cc3830fdf"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "e34e2b9267cd8f0351eed724ddbd4429"
server_key_iv: "51cbecae63be46cc9b42542823f9595e"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.210390 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:33.211068 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.211215 1025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:33.211289 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "d0bf1fa871b34d2597b0a71cc3830fdf"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "e34e2b9267cd8f0351eed724ddbd4429"
server_key_iv: "51cbecae63be46cc9b42542823f9595e"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.211349 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:33.233626 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:33.235462 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:33.235600 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:33.235805 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:33.236079 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:33.236111 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.236138 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:33.236161 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.241633 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42905
I20260924 19:31:33.241686 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42905 every 8 connection(s)
I20260924 19:31:33.241979 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:33.251111 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.242647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35633 (local address 127.0.98.62:36551)
0924 19:31:33.242702 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.242704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.242894 (+ 190us) server_negotiation.cc:407] Connection header received
0924 19:31:33.243384 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.243385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.243396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.243448 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:33.244358 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.244926 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.245896 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.246105 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.246158 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.248358 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:33.248359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:33.248359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:33.248366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:33.249633 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.250115 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.250116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.250116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.250213 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.250534 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.250535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.250535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.250675 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:33.250731 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.250965 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.251042 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:31:33.251252 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.242621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36551 (local address 127.0.98.1:35633)
0924 19:31:33.242780 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:33.242815 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:33.243293 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:33.243520 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:33.243524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:33.244075 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:33.244264 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.244268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.245019 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.245020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.245818 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.245820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.246169 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.246170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.246297 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.246766 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:33.246775 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:33.248251 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:33.249715 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:33.249721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:33.249723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:33.250015 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:33.250305 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:33.250306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:33.250306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:33.250447 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:33.250724 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:33.250729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:33.250851 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 19:31:33.250937 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:31:33.251062 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":83,"threads_started":1}
I20260924 19:31:33.251524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 1025
I20260924 19:31:33.251618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:33.251788 392 external_mini_cluster.cc:1428] Setting key c96401b84de7a5297bc4fd0ef7976e03
I20260924 19:31:33.251807 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36551
I20260924 19:31:33.251902 1154 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:33.252085 1154 heartbeater.cc:511] Master 127.0.98.62:36551 requested a full tablet report, sending...
I20260924 19:31:33.252599 962 ts_manager.cc:194] Registered new tserver with Master: d0bf1fa871b34d2597b0a71cc3830fdf (127.0.98.1:42905)
I20260924 19:31:33.253408 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35633
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:33.274916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36551
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:33.353101 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:33.353284 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:33.353305 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:33.354462 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:33.354501 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:33.354526 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:33.355894 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:33.356105 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:33.356300 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:33.361029 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:33.361070 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:33.361068 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:33.361102 1159 server_base.cc:1034] running on GCE node
I20260924 19:31:33.361356 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:33.361581 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:33.362712 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790278293362702 us; error 31 us; skew 500 ppm
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:33.364935 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:33.365329 1159 webserver.cc:460] Webserver started at http://127.0.98.2:46815/ using document root <none> and password file <none>
I20260924 19:31:33.365525 1159 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:33.365571 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:33.365686 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:33.366782 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "3688a81a748b4001880bd946b6a35879"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "d0c9196ad5f4414dc3e64338f3cc8fd7"
server_key_iv: "28c0065c9fe4e45ac26eb67f633f4e5e"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.367108 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3688a81a748b4001880bd946b6a35879"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "d0c9196ad5f4414dc3e64338f3cc8fd7"
server_key_iv: "28c0065c9fe4e45ac26eb67f633f4e5e"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.368376 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:31:33.369107 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.369243 1159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.369320 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "3688a81a748b4001880bd946b6a35879"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "d0c9196ad5f4414dc3e64338f3cc8fd7"
server_key_iv: "28c0065c9fe4e45ac26eb67f633f4e5e"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.369386 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:33.389302 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:33.390990 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:33.391119 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:33.391316 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:33.391604 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:33.391635 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.391664 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:33.391686 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.397051 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38125
I20260924 19:31:33.397122 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38125 every 8 connection(s)
I20260924 19:31:33.397382 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:31:33.399294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 1159
I20260924 19:31:33.399364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:33.399497 392 external_mini_cluster.cc:1428] Setting key fae33340ffde6b67e9cc6912d9e6a5fd
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:33.406502 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.398032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57261 (local address 127.0.98.62:36551)
0924 19:31:33.398073 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.398075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.398292 (+ 217us) server_negotiation.cc:407] Connection header received
0924 19:31:33.398777 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.398778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.398789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.398846 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:33.399832 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.400326 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.401483 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.401678 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.401736 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.403815 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:33.403816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:33.403817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:33.403825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:33.405272 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.405622 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.405623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.405624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.405683 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.405891 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.405892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.405892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.406027 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:33.406102 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.406328 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.406402 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:33.406598 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.397990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36551 (local address 127.0.98.2:57261)
0924 19:31:33.398186 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:33.398219 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:33.398690 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:33.398919 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:33.398924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:33.399489 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:33.399754 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.399760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.400405 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.400407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.401406 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.401409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.401754 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.401755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.401888 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.402381 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:33.402389 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:33.403729 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:33.405342 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:33.405347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:33.405349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:33.405553 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:33.405744 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:33.405744 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:33.405745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:33.405835 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:33.406092 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:33.406095 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:33.406207 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:31:33.406285 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:31:33.406401 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
I20260924 19:31:33.407102 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36551
I20260924 19:31:33.407188 1288 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:31:33.407359 1288 heartbeater.cc:511] Master 127.0.98.62:36551 requested a full tablet report, sending...
I20260924 19:31:33.407720 962 ts_manager.cc:194] Registered new tserver with Master: 3688a81a748b4001880bd946b6a35879 (127.0.98.2:38125)
I20260924 19:31:33.408082 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:33.423602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36551
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:33.501639 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:33.501832 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:33.501852 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:33.502935 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:33.502969 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:33.502992 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:33.504323 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36551
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:33.504492 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:33.504684 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:33.509230 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:33.509341 1293 server_base.cc:1034] running on GCE node
W20260924 19:31:33.509240 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:33.509240 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:33.509593 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:33.509799 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:33.510938 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790278293510920 us; error 31 us; skew 500 ppm
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:33.513141 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:33.513567 1293 webserver.cc:460] Webserver started at http://127.0.98.3:41185/ using document root <none> and password file <none>
I20260924 19:31:33.513764 1293 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:33.513810 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:33.513931 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:33.514946 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "396cb29be7251a87f516d57f97beb92f"
server_key_iv: "21effe1d9cc521f5413cab781e9576d3"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.515270 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "396cb29be7251a87f516d57f97beb92f"
server_key_iv: "21effe1d9cc521f5413cab781e9576d3"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.516434 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:33.517196 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.517335 1293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.517411 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd"
format_stamp: "Formatted at 2026-09-24 19:31:33 on dist-test-slave-m8jb"
server_key: "396cb29be7251a87f516d57f97beb92f"
server_key_iv: "21effe1d9cc521f5413cab781e9576d3"
server_key_version: "encryptionkey@0"
I20260924 19:31:33.517475 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:33.534582 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:33.536361 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:33.536492 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:33.536774 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:33.537070 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:33.537102 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.537130 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:33.537153 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:33.542464 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33379
I20260924 19:31:33.542532 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33379 every 8 connection(s)
I20260924 19:31:33.542819 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:31:33.548026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 1293
I20260924 19:31:33.548113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:33.548254 392 external_mini_cluster.cc:1428] Setting key 134698b1cd0f30addf3cff55bd949305
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:33.550925 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.543479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34849 (local address 127.0.98.62:36551)
0924 19:31:33.543535 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.543537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.543734 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:31:33.544187 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.544188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.544197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.544249 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:33.545154 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.545648 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.546574 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.546758 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.546809 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.548884 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:33.548885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:33.548885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:33.548892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:33.549945 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.550265 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.550266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.550266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.550308 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.550473 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.550473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.550473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.550562 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:33.550605 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.550810 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.550863 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:31:33.551112 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.543455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36551 (local address 127.0.98.3:34849)
0924 19:31:33.543615 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:33.543655 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:33.544115 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:33.544306 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:33.544309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:33.544884 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:33.545077 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.545081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.545712 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.545712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.546507 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.546509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.546817 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.546818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.546910 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.547373 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:33.547381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:33.548814 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:33.550009 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:33.550015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:33.550016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:33.550212 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:33.550349 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:33.550350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:33.550350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:33.550433 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:33.550605 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:33.550608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:33.550718 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 19:31:33.550796 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 19:31:33.550905 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260924 19:31:33.551452 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36551
I20260924 19:31:33.551523 1422 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:33.551654 1422 heartbeater.cc:511] Master 127.0.98.62:36551 requested a full tablet report, sending...
I20260924 19:31:33.551945 962 ts_manager.cc:194] Registered new tserver with Master: a6e7e25677fd4b80b56a5b0e70c1c4fd (127.0.98.3:33379)
I20260924 19:31:33.552269 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34849
I20260924 19:31:33.559949 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:31:33.567170 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.561499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.0.98.62:36551)
0924 19:31:33.561543 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.561545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.561668 (+ 123us) server_negotiation.cc:407] Connection header received
0924 19:31:33.561712 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.561713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.561720 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.561767 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:33.562286 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.562785 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.563481 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.563675 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.563726 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.564920 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:33.564921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:33.564922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:33.564926 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:33.566091 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.566421 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.566422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.566422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.566488 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.566701 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.566702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.566703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.566863 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:33.566956 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.567029 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.567093 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:31:33.569249 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:33.570847 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:33.576815 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.572808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38125 (local address 127.0.0.1:37428)
0924 19:31:33.573096 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:33.573107 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:33.573156 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:33.573465 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:33.573467 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:33.573470 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:33.573534 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.573536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.574832 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.574833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.575667 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.575669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.576535 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.576536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.576610 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.576626 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 19:31:33.576692 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:31:33.576742 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":65,"threads_started":1}
I20260924 19:31:33.576895 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.572805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42905 (local address 127.0.0.1:56622)
0924 19:31:33.572988 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:33.573022 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:33.573084 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:33.573650 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:33.573657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:33.573660 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:33.573707 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.573709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.574956 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.574957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.575987 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.575989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.576671 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.576672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.576732 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.576736 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:33.576782 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:31:33.576821 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":54,"threads_started":1}
I20260924 19:31:33.576982 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.572923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.0.98.1:42905)
0924 19:31:33.573488 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.573501 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.573509 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:33.573542 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.573544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.573556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.573778 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:33.573847 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.574869 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.576051 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.576581 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.576669 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.576729 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.576862 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.576907 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":90,"threads_started":1}
I20260924 19:31:33.576987 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.573129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.0.98.3:33379)
0924 19:31:33.573282 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.573286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.573362 (+ 76us) server_negotiation.cc:407] Connection header received
0924 19:31:33.573439 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.573442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.573456 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.573530 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:33.573848 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.574541 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.575908 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.576611 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.576704 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.576757 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.576877 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.576915 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260924 19:31:33.577242 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.573139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.0.98.2:38125)
0924 19:31:33.573290 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.573295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.573312 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:31:33.573354 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.573357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.573372 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.573448 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:33.573602 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.574724 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.575743 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.576430 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.577072 (+ 642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.577116 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.577136 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.577165 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260924 19:31:33.577591 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.573026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33379 (local address 127.0.0.1:43552)
0924 19:31:33.573310 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:33.573318 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:33.573370 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:33.573616 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:33.573617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:33.573619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:33.573719 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.573722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.574637 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.574638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.575828 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.575830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.576757 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.576758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.576809 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.576811 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:31:33.576858 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:31:33.576891 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":30,"threads_started":1}
I20260924 19:31:33.577826 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 6de59fb16b004e9a8f867cdea01105d8 (DEFAULT_TABLE table=test-table [id=a30e59dc9ea7402dbfe67509b3fa7e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:33.577893 1222 tablet_service.cc:1467] Processing CreateTablet for tablet 6de59fb16b004e9a8f867cdea01105d8 (DEFAULT_TABLE table=test-table [id=a30e59dc9ea7402dbfe67509b3fa7e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:33.578173 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de59fb16b004e9a8f867cdea01105d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:33.578190 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de59fb16b004e9a8f867cdea01105d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:33.578207 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 6de59fb16b004e9a8f867cdea01105d8 (DEFAULT_TABLE table=test-table [id=a30e59dc9ea7402dbfe67509b3fa7e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:33.578475 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6de59fb16b004e9a8f867cdea01105d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:33.581331 1436 tablet_bootstrap.cc:492] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879: Bootstrap starting.
I20260924 19:31:33.581372 1437 tablet_bootstrap.cc:492] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf: Bootstrap starting.
I20260924 19:31:33.581372 1438 tablet_bootstrap.cc:492] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd: Bootstrap starting.
I20260924 19:31:33.581936 1436 tablet_bootstrap.cc:654] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:33.581971 1438 tablet_bootstrap.cc:654] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:33.581971 1437 tablet_bootstrap.cc:654] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:33.582383 1436 log.cc:824] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:33.582446 1437 log.cc:824] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:33.582446 1438 log.cc:824] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:33.583087 1436 tablet_bootstrap.cc:492] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879: No bootstrap required, opened a new log
I20260924 19:31:33.583094 1438 tablet_bootstrap.cc:492] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd: No bootstrap required, opened a new log
I20260924 19:31:33.583156 1436 ts_tablet_manager.cc:1397] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:33.583156 1438 ts_tablet_manager.cc:1397] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:33.583232 1437 tablet_bootstrap.cc:492] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf: No bootstrap required, opened a new log
I20260924 19:31:33.583294 1437 ts_tablet_manager.cc:1397] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:33.585088 1437 raft_consensus.cc:348] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.585230 1437 raft_consensus.cc:374] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:33.585129 1436 raft_consensus.cc:348] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.585261 1437 raft_consensus.cc:729] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0bf1fa871b34d2597b0a71cc3830fdf, State: Initialized, Role: FOLLOWER
I20260924 19:31:33.585300 1436 raft_consensus.cc:374] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:33.585332 1436 raft_consensus.cc:729] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3688a81a748b4001880bd946b6a35879, State: Initialized, Role: FOLLOWER
I20260924 19:31:33.585405 1437 consensus_queue.cc:260] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.585433 1436 consensus_queue.cc:260] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.585471 1438 raft_consensus.cc:348] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.585629 1438 raft_consensus.cc:374] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:33.585675 1438 raft_consensus.cc:729] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e7e25677fd4b80b56a5b0e70c1c4fd, State: Initialized, Role: FOLLOWER
I20260924 19:31:33.585685 1436 ts_tablet_manager.cc:1428] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:31:33.585685 1437 ts_tablet_manager.cc:1428] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:31:33.585778 1438 consensus_queue.cc:260] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.585817 1288 heartbeater.cc:503] Master 127.0.98.62:36551 was elected leader, sending a full tablet report...
I20260924 19:31:33.585820 1154 heartbeater.cc:503] Master 127.0.98.62:36551 was elected leader, sending a full tablet report...
I20260924 19:31:33.586005 1438 ts_tablet_manager.cc:1428] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:31:33.586174 1422 heartbeater.cc:503] Master 127.0.98.62:36551 was elected leader, sending a full tablet report...
W20260924 19:31:33.647965 1289 tablet.cc:2369] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:31:33.717283 1444 raft_consensus.cc:482] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:33.717393 1444 raft_consensus.cc:504] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.717736 1444 leader_election.cc:288] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0bf1fa871b34d2597b0a71cc3830fdf (127.0.98.1:42905), 3688a81a748b4001880bd946b6a35879 (127.0.98.2:38125)
I20260924 19:31:33.720518 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.717916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38125 (local address 127.0.98.3:33837)
0924 19:31:33.717961 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:33.717971 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:33.718010 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:33.718174 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:33.718175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:33.718176 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:33.718242 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.718244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.718928 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.718929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.719770 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.719773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.720287 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.720288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.720362 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.720366 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:33.720422 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:31:33.720459 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":17}
I20260924 19:31:33.720534 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.717969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33837 (local address 127.0.98.2:38125)
0924 19:31:33.718003 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.718005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.718014 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:33.718088 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.718089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.718101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.718166 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:33.718307 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.718849 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.719865 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.720228 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.720284 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.720295 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.720429 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.720461 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:33.720678 1445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.717916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42905 (local address 127.0.98.3:41755)
0924 19:31:33.718145 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:33.718155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:33.718209 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:33.718318 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:33.718319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:33.718320 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:33.718379 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.718380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.718987 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.718988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.719769 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:33.719771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.720437 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.720438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:33.720500 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.720503 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:33.720555 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:31:33.720589 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":62,"thread_start_us":92,"threads_started":1}
I20260924 19:31:33.720685 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.717957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41755 (local address 127.0.98.1:42905)
0924 19:31:33.717999 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.718018 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.718216 (+ 198us) server_negotiation.cc:407] Connection header received
0924 19:31:33.718247 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.718248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.718261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.718312 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:33.718453 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.718911 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.719852 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.720372 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.720430 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.720439 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.720568 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.720607 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:31:33.720978 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de59fb16b004e9a8f867cdea01105d8" candidate_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" is_pre_election: true
I20260924 19:31:33.721141 1108 raft_consensus.cc:2454] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e7e25677fd4b80b56a5b0e70c1c4fd in term 0.
I20260924 19:31:33.721203 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de59fb16b004e9a8f867cdea01105d8" candidate_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3688a81a748b4001880bd946b6a35879" is_pre_election: true
I20260924 19:31:33.721346 1242 raft_consensus.cc:2454] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e7e25677fd4b80b56a5b0e70c1c4fd in term 0.
I20260924 19:31:33.721345 1312 leader_election.cc:302] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e7e25677fd4b80b56a5b0e70c1c4fd, d0bf1fa871b34d2597b0a71cc3830fdf; no voters:
I20260924 19:31:33.721462 1444 raft_consensus.cc:2781] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:33.721506 1444 raft_consensus.cc:482] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:33.721526 1444 raft_consensus.cc:3037] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:33.722170 1444 raft_consensus.cc:504] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.722280 1444 leader_election.cc:288] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [CANDIDATE]: Term 1 election: Requested vote from peers d0bf1fa871b34d2597b0a71cc3830fdf (127.0.98.1:42905), 3688a81a748b4001880bd946b6a35879 (127.0.98.2:38125)
I20260924 19:31:33.722415 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de59fb16b004e9a8f867cdea01105d8" candidate_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3688a81a748b4001880bd946b6a35879"
I20260924 19:31:33.722487 1242 raft_consensus.cc:3037] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:33.723166 1242 raft_consensus.cc:2454] T 6de59fb16b004e9a8f867cdea01105d8 P 3688a81a748b4001880bd946b6a35879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e7e25677fd4b80b56a5b0e70c1c4fd in term 1.
I20260924 19:31:33.723259 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6de59fb16b004e9a8f867cdea01105d8" candidate_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf"
I20260924 19:31:33.723320 1310 leader_election.cc:302] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3688a81a748b4001880bd946b6a35879, a6e7e25677fd4b80b56a5b0e70c1c4fd; no voters:
I20260924 19:31:33.723337 1108 raft_consensus.cc:3037] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:33.723393 1444 raft_consensus.cc:2781] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:33.723534 1444 raft_consensus.cc:686] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [term 1 LEADER]: Becoming Leader. State: Replica: a6e7e25677fd4b80b56a5b0e70c1c4fd, State: Running, Role: LEADER
I20260924 19:31:33.723651 1444 consensus_queue.cc:237] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } }
I20260924 19:31:33.724128 1108 raft_consensus.cc:2454] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e7e25677fd4b80b56a5b0e70c1c4fd in term 1.
I20260924 19:31:33.724422 960 catalog_manager.cc:5495] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e7e25677fd4b80b56a5b0e70c1c4fd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bf1fa871b34d2597b0a71cc3830fdf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3688a81a748b4001880bd946b6a35879" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e7e25677fd4b80b56a5b0e70c1c4fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33379 } health_report { overall_health: HEALTHY } } }
W20260924 19:31:33.742650 1155 tablet.cc:2369] T 6de59fb16b004e9a8f867cdea01105d8 P d0bf1fa871b34d2597b0a71cc3830fdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:33.760412 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.753797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.1:42905)
0924 19:31:33.753856 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.753858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.753941 (+ 83us) server_negotiation.cc:407] Connection header received
0924 19:31:33.753999 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.754000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.754028 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.754073 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:33.754604 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.755071 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.755747 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.755925 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.755978 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.758063 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:33.758077 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:33.758077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:33.758083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:33.759344 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.759678 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.759682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.759682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.759745 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.759909 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.759910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.759910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.760051 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:33.760183 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.760243 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.760320 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:33.781306 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.775276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.98.1:42905)
0924 19:31:33.775339 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.775341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.775390 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:31:33.775447 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.775448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.775456 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.775509 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:33.775998 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.776463 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.777189 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.777368 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.777409 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.779365 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:33.779367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:33.779367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:33.779373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:33.780423 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.780749 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.780750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.780750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.780797 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.780921 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.780921 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.780921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.781000 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:33.781061 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.781175 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.781232 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260924 19:31:33.793308 1423 tablet.cc:2369] T 6de59fb16b004e9a8f867cdea01105d8 P a6e7e25677fd4b80b56a5b0e70c1c4fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:33.802004 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:33.795115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.0.98.1:42905)
0924 19:31:33.795169 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:33.795171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:33.795645 (+ 474us) server_negotiation.cc:407] Connection header received
0924 19:31:33.795712 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:33.795713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:33.795724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:33.795768 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:33.796260 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.796778 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.797481 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:33.797666 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:33.797718 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:33.799981 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:33.799983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:33.799983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:33.799990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:33.801067 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.801353 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.801354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.801354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.801403 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:33.801564 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:33.801564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:33.801565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:33.801660 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:33.801749 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:33.801841 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 19:31:33.801913 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 19:31:33.802467 1068 tablet_service.cc:2908] Rejecting scan request for tablet 6de59fb16b004e9a8f867cdea01105d8: Uninitialized: clean time has not yet been initialized
I20260924 19:31:33.803372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 1025
I20260924 19:31:33.808014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 1159
I20260924 19:31:33.811493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 1293
I20260924 19:31:33.815433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 930
2026-09-24T19:31:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1940 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[1460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[1460](info): set up 2 sockets
Sep 24 19:31:33 dist-test-slave-m8jb krb5kdc[1460](info): commencing operation
krb5kdc: starting...
W20260924 19:31:34.763917 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:34 dist-test-slave-m8jb krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:34.875388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42785
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:34.953194 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:34.953383 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:34.953403 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:34.954533 1473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:34.954568 1473 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:31:34.954579 1473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:34.954591 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:34.954602 1473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:34.954613 1473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:34.956040 1473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42785
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:34.956216 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:34.956383 1473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:34.960801 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:34.960794 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:34.960795 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:34.961057 1473 server_base.cc:1034] running on GCE node
I20260924 19:31:34.961269 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:34.961552 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:34.962702 1473 hybrid_clock.cc:648] HybridClock initialized: now 1790278294962689 us; error 30 us; skew 500 ppm
Sep 24 19:31:34 dist-test-slave-m8jb krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:34.965029 1473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:34.965436 1473 webserver.cc:460] Webserver started at http://127.0.98.62:41883/ using document root <none> and password file <none>
I20260924 19:31:34.965685 1473 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:34.965731 1473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:34.965853 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:34.966854 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "54a1e25f156a49549be1521fabdc6bed"
format_stamp: "Formatted at 2026-09-24 19:31:34 on dist-test-slave-m8jb"
server_key: "a64ba8f200e050250cbde1a6f107de60"
server_key_iv: "89fc96f95250af8065aad4ddf726ce00"
server_key_version: "encryptionkey@0"
I20260924 19:31:34.967176 1473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "54a1e25f156a49549be1521fabdc6bed"
format_stamp: "Formatted at 2026-09-24 19:31:34 on dist-test-slave-m8jb"
server_key: "a64ba8f200e050250cbde1a6f107de60"
server_key_iv: "89fc96f95250af8065aad4ddf726ce00"
server_key_version: "encryptionkey@0"
I20260924 19:31:34.968338 1473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:31:34.968971 1488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:34.969110 1473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:34.969189 1473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "54a1e25f156a49549be1521fabdc6bed"
format_stamp: "Formatted at 2026-09-24 19:31:34 on dist-test-slave-m8jb"
server_key: "a64ba8f200e050250cbde1a6f107de60"
server_key_iv: "89fc96f95250af8065aad4ddf726ce00"
server_key_version: "encryptionkey@0"
I20260924 19:31:34.969261 1473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:34.993796 1473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:34.995931 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:34.996068 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:35.000481 1473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42785
I20260924 19:31:35.000521 1550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42785 every 8 connection(s)
I20260924 19:31:35.000870 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:35.001473 1551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.003670 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed: Bootstrap starting.
I20260924 19:31:35.004222 1551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.004477 1551 log.cc:824] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:35.005199 1551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed: No bootstrap required, opened a new log
I20260924 19:31:35.007179 1551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a1e25f156a49549be1521fabdc6bed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42785 } }
I20260924 19:31:35.007309 1551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.007330 1551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a1e25f156a49549be1521fabdc6bed, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.007455 1551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a1e25f156a49549be1521fabdc6bed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42785 } }
I20260924 19:31:35.007524 1551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:35.007544 1551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:35.007586 1551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:35.008062 1551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a1e25f156a49549be1521fabdc6bed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42785 } }
I20260924 19:31:35.008150 1551 leader_election.cc:302] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54a1e25f156a49549be1521fabdc6bed; no voters:
I20260924 19:31:35.008291 1551 leader_election.cc:288] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:35.008350 1554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:35.008525 1554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [term 1 LEADER]: Becoming Leader. State: Replica: 54a1e25f156a49549be1521fabdc6bed, State: Running, Role: LEADER
I20260924 19:31:35.008533 1551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:35.008697 1554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a1e25f156a49549be1521fabdc6bed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42785 } }
I20260924 19:31:35.009033 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54a1e25f156a49549be1521fabdc6bed. Latest consensus state: current_term: 1 leader_uuid: "54a1e25f156a49549be1521fabdc6bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a1e25f156a49549be1521fabdc6bed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42785 } } }
I20260924 19:31:35.009135 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:35.009300 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54a1e25f156a49549be1521fabdc6bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54a1e25f156a49549be1521fabdc6bed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42785 } } }
I20260924 19:31:35.009320 1563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:35.009364 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:35.009835 1563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:35.009940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 1473
I20260924 19:31:35.010043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:35.010205 392 external_mini_cluster.cc:1428] Setting key 8c6182d82aca7a0f2697cb8cdb2df44a
I20260924 19:31:35.011478 1563 catalog_manager.cc:1347] Generated new cluster ID: 0b9851c1584040aaabf0ca0cec1d3982
I20260924 19:31:35.011528 1563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:35.021940 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.010631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54808 (local address 127.0.98.62:42785)
0924 19:31:35.010772 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.010781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.011034 (+ 253us) server_negotiation.cc:407] Connection header received
0924 19:31:35.011156 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.011162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.011206 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.011316 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.012023 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.013498 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.014499 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.015187 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.015299 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.018297 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.018302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.018302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.018314 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.020234 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.020627 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.020657 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.020658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.020756 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.020938 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.020939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.020939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.021217 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.021366 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.021459 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.021697 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260924 19:31:35.023365 1563 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:35.023876 1563 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:35.029623 1563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed: Generated new TSK 0
I20260924 19:31:35.029768 1563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:35.056219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42785
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:35.135031 1577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:35.135238 1577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:35.135269 1577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:35.136410 1577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:35.136451 1577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:35.136478 1577 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:31:35.136500 1577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:35.137980 1577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:35.138222 1577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:35.138420 1577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:35.143189 1583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:35.143257 1582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:35.143292 1585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:35.143349 1577 server_base.cc:1034] running on GCE node
I20260924 19:31:35.143530 1577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:35.143743 1577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:35.144870 1577 hybrid_clock.cc:648] HybridClock initialized: now 1790278295144858 us; error 27 us; skew 500 ppm
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:35.147215 1577 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:35.147622 1577 webserver.cc:460] Webserver started at http://127.0.98.1:33535/ using document root <none> and password file <none>
I20260924 19:31:35.147794 1577 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:35.147832 1577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:35.147917 1577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:35.148952 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "6a0840976c894de6a0bce37c896dc810"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "26b2d1d28eef26ffbc40d9bc17388d3f"
server_key_iv: "1184b47ba50e2b7a17a9ff0d4543482e"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.149260 1577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6a0840976c894de6a0bce37c896dc810"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "26b2d1d28eef26ffbc40d9bc17388d3f"
server_key_iv: "1184b47ba50e2b7a17a9ff0d4543482e"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.150460 1577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:35.151095 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.151221 1577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:35.151290 1577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "6a0840976c894de6a0bce37c896dc810"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "26b2d1d28eef26ffbc40d9bc17388d3f"
server_key_iv: "1184b47ba50e2b7a17a9ff0d4543482e"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.151335 1577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:35.167609 1577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:35.169706 1577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:35.169834 1577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:35.170356 1577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:35.170397 1577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.170426 1577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:35.170449 1577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.176738 1577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43623
I20260924 19:31:35.176844 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43623 every 8 connection(s)
I20260924 19:31:35.177150 1577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:35.180768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 1577
I20260924 19:31:35.180867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:35.181044 392 external_mini_cluster.cc:1428] Setting key 0c98fbf8a4c50cd5966af3963d12a715
I20260924 19:31:35.182457 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.170750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39153 (local address 127.0.98.62:42785)
0924 19:31:35.170816 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.170819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.170974 (+ 155us) server_negotiation.cc:407] Connection header received
0924 19:31:35.171461 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.171463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.171475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.171549 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.172346 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.173076 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.174083 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.174350 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.174401 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.179740 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.179741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.179742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.179751 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.181159 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.181589 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.181590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.181590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.181651 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.181877 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.181877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.181878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.182002 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.182072 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.182296 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.182374 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:31:35.182565 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.170716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42785 (local address 127.0.98.1:39153)
0924 19:31:35.170856 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.170896 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.171374 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.171629 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.171634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.171969 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:35.172263 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.172269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.173150 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.173151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.173997 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.173999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.174437 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.174438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.174545 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.178330 (+ 3785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:35.178338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:35.179629 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:35.181232 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.181240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.181242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.181521 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.181708 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.181708 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.181709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.181820 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.182061 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:35.182064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:35.182188 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.182267 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.182388 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260924 19:31:35.182998 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42785
I20260924 19:31:35.183076 1708 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:35.183265 1708 heartbeater.cc:511] Master 127.0.98.62:42785 requested a full tablet report, sending...
I20260924 19:31:35.183820 1504 ts_manager.cc:194] Registered new tserver with Master: 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623)
I20260924 19:31:35.184697 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39153
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 19:31:35.189069 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.183924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42785 (local address 127.0.98.1:48939)
0924 19:31:35.184070 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.184080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.184131 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.184240 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.184240 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.184391 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:35.184444 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.184445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.185226 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.185227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.185765 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.185767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.186100 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.186101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.186169 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.186578 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:35.186582 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:35.186854 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:35.188155 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.188156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.188157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.188308 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.188468 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.188469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.188469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.188523 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.188797 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:35.188798 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:35.188841 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.188914 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.188996 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":25,"thread_start_us":64,"threads_started":1}
I20260924 19:31:35.189113 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.183980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48939 (local address 127.0.98.62:42785)
0924 19:31:35.184019 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.184021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.184139 (+ 118us) server_negotiation.cc:407] Connection header received
0924 19:31:35.184179 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.184180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.184190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.184228 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.184505 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.185110 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.185843 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.186041 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.186095 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.186930 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.186931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.186931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.186937 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.188079 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.188365 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.188366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.188366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.188417 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.188571 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.188572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.188572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.188717 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.188770 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.188975 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.189054 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:35.205305 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42785
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:35.282787 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:35.282968 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:35.282990 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:35.284125 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:35.284162 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:35.284180 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:31:35.284209 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:35.285672 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:35.285853 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:35.286028 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:35.290829 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:35.290829 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:35.290829 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:35.291054 1717 server_base.cc:1034] running on GCE node
I20260924 19:31:35.291234 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:35.291441 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:35.292587 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790278295292575 us; error 28 us; skew 500 ppm
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:35.294992 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:35.295533 1717 webserver.cc:460] Webserver started at http://127.0.98.2:36659/ using document root <none> and password file <none>
I20260924 19:31:35.295809 1717 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:35.295876 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:35.296008 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:35.297149 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "abbe26b5345e4e458808231c5e94a6b6"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "12d98ae9e9651f32b84b5576fe980ffa"
server_key_iv: "9ff2ca9867204da6c46cf353f081fd31"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.297478 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "abbe26b5345e4e458808231c5e94a6b6"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "12d98ae9e9651f32b84b5576fe980ffa"
server_key_iv: "9ff2ca9867204da6c46cf353f081fd31"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.299129 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:35.299959 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.300109 1717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:35.300207 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "abbe26b5345e4e458808231c5e94a6b6"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "12d98ae9e9651f32b84b5576fe980ffa"
server_key_iv: "9ff2ca9867204da6c46cf353f081fd31"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.300287 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:35.312739 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:35.314854 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:35.314975 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:35.315533 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:35.315586 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.315620 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:35.315647 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:35.322901 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42753
I20260924 19:31:35.323264 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:31:35.323709 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42753 every 8 connection(s)
I20260924 19:31:35.324689 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.315930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59407 (local address 127.0.98.62:42785)
0924 19:31:35.315993 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.315995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.316172 (+ 177us) server_negotiation.cc:407] Connection header received
0924 19:31:35.316666 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.316668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.316682 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.316766 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.317346 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.318078 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.319128 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.319374 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.319426 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.322012 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.322013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.322014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.322022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.323365 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.323807 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.323808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.323809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.323872 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.324088 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.324088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.324088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.324191 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.324237 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.324514 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.324574 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:31:35.324865 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.315909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42785 (local address 127.0.98.2:59407)
0924 19:31:35.316043 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.316083 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.316553 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.316821 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.316825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.317053 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:35.317270 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.317275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.318195 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.318196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.319044 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.319046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.319451 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.319452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.319565 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.320005 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:35.320014 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:35.321908 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:35.323430 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.323435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.323437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.323737 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.323940 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.323941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.323941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.324039 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.324266 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:35.324272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:35.324401 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.324498 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.324650 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":70,"threads_started":1}
I20260924 19:31:35.325280 1848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42785
I20260924 19:31:35.325356 1848 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:35.325528 1848 heartbeater.cc:511] Master 127.0.98.62:42785 requested a full tablet report, sending...
I20260924 19:31:35.325899 1505 ts_manager.cc:194] Registered new tserver with Master: abbe26b5345e4e458808231c5e94a6b6 (127.0.98.2:42753)
I20260924 19:31:35.326339 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59407
I20260924 19:31:35.329507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 1717
I20260924 19:31:35.329599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:35.329779 392 external_mini_cluster.cc:1428] Setting key 38f3a0c3c34f351892617f5cd4b225d0
I20260924 19:31:35.331101 1854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.326171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42785 (local address 127.0.98.2:35577)
0924 19:31:35.326282 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.326291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.326340 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.326493 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.326494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.326667 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:35.326736 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.326738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.327363 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.327363 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.327867 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.327868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.328128 (+ 260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.328129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.328168 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.328477 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:35.328481 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:35.328769 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:35.330078 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.330079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.330079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.330266 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.330524 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.330524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.330525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.330575 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.330875 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:35.330876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:35.330906 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.330949 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.331017 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260924 19:31:35.331148 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.326246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35577 (local address 127.0.98.62:42785)
0924 19:31:35.326286 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.326288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.326327 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:31:35.326381 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.326382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.326391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.326433 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.326820 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.327284 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.327921 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.328083 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.328133 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.328830 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.328832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.328832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.328837 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.330009 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.330353 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.330354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.330354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.330457 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.330630 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.330630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.330631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.330809 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.330881 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.330992 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.331068 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:35.353282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42785
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:35.431354 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:35.431545 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:35.431576 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:35.432715 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:35.432754 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:35.432780 1857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:31:35.432803 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:35.434273 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:35.434492 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:35.434700 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:35.439283 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:35.439260 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:35.439260 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:35.439476 1857 server_base.cc:1034] running on GCE node
I20260924 19:31:35.439663 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:35.439873 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:35.440999 1857 hybrid_clock.cc:648] HybridClock initialized: now 1790278295440987 us; error 28 us; skew 500 ppm
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:35.443226 1857 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:35.443632 1857 webserver.cc:460] Webserver started at http://127.0.98.3:42559/ using document root <none> and password file <none>
I20260924 19:31:35.443817 1857 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:35.443862 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:35.443976 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:35.445032 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "e231a9acaccb4811ba1caf4f7d52bac4"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "b0b879b5b1b305d43b5d9995e7700cb9"
server_key_iv: "79d2d46267233b1cd53a9746f2e29021"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.445341 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e231a9acaccb4811ba1caf4f7d52bac4"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "b0b879b5b1b305d43b5d9995e7700cb9"
server_key_iv: "79d2d46267233b1cd53a9746f2e29021"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.446914 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:35.447535 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.447679 1857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:35.447763 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "e231a9acaccb4811ba1caf4f7d52bac4"
format_stamp: "Formatted at 2026-09-24 19:31:35 on dist-test-slave-m8jb"
server_key: "b0b879b5b1b305d43b5d9995e7700cb9"
server_key_iv: "79d2d46267233b1cd53a9746f2e29021"
server_key_version: "encryptionkey@0"
I20260924 19:31:35.447817 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:35.460738 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:35.462700 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:35.462844 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:35.463411 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:35.463459 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.463613 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:35.463656 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:35.470250 1857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40263
I20260924 19:31:35.470319 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40263 every 8 connection(s)
I20260924 19:31:35.470592 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:31:35.472975 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.463868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57621 (local address 127.0.98.62:42785)
0924 19:31:35.463916 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.463918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.464112 (+ 194us) server_negotiation.cc:407] Connection header received
0924 19:31:35.464799 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.464801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.464812 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.464853 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.465664 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.466411 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.467928 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.468162 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.468224 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.470396 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.470397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.470398 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.470405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.471822 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.472170 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.472171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.472171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.472219 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.472373 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.472374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.472374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.472478 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.472533 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.472848 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.472913 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:31:35.473109 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.463839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42785 (local address 127.0.98.3:57621)
0924 19:31:35.463980 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.464020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.464701 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.464929 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.464933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.465266 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:35.465570 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.465577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.466520 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.466521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.467833 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.467836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.468248 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.468249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.468393 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.468926 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:35.468936 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:35.470292 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:35.471880 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.471884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.471885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.472106 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.472259 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.472259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.472260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.472342 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.472552 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:35.472556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:35.472714 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.472814 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.472931 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260924 19:31:35.473555 1988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42785
I20260924 19:31:35.473632 1988 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:35.473790 1988 heartbeater.cc:511] Master 127.0.98.62:42785 requested a full tablet report, sending...
I20260924 19:31:35.474129 1505 ts_manager.cc:194] Registered new tserver with Master: e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263)
I20260924 19:31:35.474555 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57621
I20260924 19:31:35.478193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 1857
I20260924 19:31:35.478294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:35.478454 392 external_mini_cluster.cc:1428] Setting key 9a92539f9b992ffe1177b3bfcd5a2693
I20260924 19:31:35.479213 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:31:35.479385 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.474530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55835 (local address 127.0.98.62:42785)
0924 19:31:35.474571 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.474573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.474637 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:31:35.474693 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.474694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.474703 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.474751 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.475009 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.475463 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.476143 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.476334 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.476384 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.477189 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.477190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.477190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.477196 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.478425 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.478737 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.478738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.478739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.478791 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.478945 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.478946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.478946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.479048 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.479100 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.479227 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.479294 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:31:35.479398 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.474469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42785 (local address 127.0.98.3:55835)
0924 19:31:35.474580 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.474589 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.474648 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.474753 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.474753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.474887 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:35.474943 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.474944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.475547 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.475548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.476076 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.476077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.476388 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.476389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.476446 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.476837 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:35.476841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:35.477115 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:35.478500 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.478501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.478501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.478676 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.478842 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.478843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.478843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.478898 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.479099 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:35.479100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:35.479142 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.479201 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.479296 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":61,"threads_started":1}
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:35.509298 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.501488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.0.98.62:42785)
0924 19:31:35.501554 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.501556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.501567 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:31:35.501599 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.501600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.501612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.501697 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.502554 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.503303 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.504329 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.504582 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.504699 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.507140 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.507142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.507142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.507150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.508297 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.508650 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.508652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.508652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.508715 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.508864 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.508864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.508864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.508959 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.509021 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.509155 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.509211 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:31:35.510921 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:35.512460 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:35.518580 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.514416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43623 (local address 127.0.0.1:56880)
0924 19:31:35.514783 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.514791 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.514839 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.515014 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.515015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.515016 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:35.515119 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.515122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.516378 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.516379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.517636 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.517637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.518297 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.518298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.518381 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.518395 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.518457 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.518505 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":100,"thread_start_us":51,"threads_started":1}
I20260924 19:31:35.518606 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.514594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.0.98.1:43623)
0924 19:31:35.514779 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.514783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.514876 (+ 93us) server_negotiation.cc:407] Connection header received
0924 19:31:35.514911 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.514913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.514926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.514995 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:35.515179 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.516291 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.517712 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.518224 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.518286 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.518366 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.518468 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.518507 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20260924 19:31:35.519337 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.514420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40263 (local address 127.0.0.1:49676)
0924 19:31:35.514878 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.514884 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.514906 (+ 22us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.515251 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.515252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.515253 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:35.515282 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.515284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.516441 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.516442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.517689 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.517692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.519118 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.519119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.519176 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.519181 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.519230 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.519267 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":27,"thread_start_us":59,"threads_started":1}
I20260924 19:31:35.519449 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5802eb8b364735846bcccfe2f14a1f (DEFAULT_TABLE table=test-table [id=a852d0885c7c44c1806d18fbfc82bf08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:35.519743 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.514645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.0.98.3:40263)
0924 19:31:35.515115 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.515119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.515127 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:35.515159 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.515161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.515174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.515255 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:35.515332 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.516358 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.517754 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.518486 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.518551 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.518619 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.519444 (+ 825us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.519488 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":64,"threads_started":1}
I20260924 19:31:35.519759 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.514416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42753 (local address 127.0.0.1:36026)
0924 19:31:35.514574 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.514613 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.514676 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.515007 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.515010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.515012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:35.515119 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.515128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.517559 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.517560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.518618 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.518621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.519578 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.519579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.519639 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.519643 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.519671 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.519703 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":28,"thread_start_us":52,"threads_started":1}
I20260924 19:31:35.519838 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5802eb8b364735846bcccfe2f14a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.520061 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.514578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36026 (local address 127.0.98.2:42753)
0924 19:31:35.514845 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.514848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.514857 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:35.514896 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.514898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.514913 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.514995 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:35.516388 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.517481 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.518703 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.519374 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.519851 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.519935 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.519960 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.519993 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
I20260924 19:31:35.520682 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5802eb8b364735846bcccfe2f14a1f (DEFAULT_TABLE table=test-table [id=a852d0885c7c44c1806d18fbfc82bf08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:35.521004 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5802eb8b364735846bcccfe2f14a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.523656 2008 tablet_bootstrap.cc:492] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Bootstrap starting.
I20260924 19:31:35.524266 2008 tablet_bootstrap.cc:654] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.524259 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 8d5802eb8b364735846bcccfe2f14a1f (DEFAULT_TABLE table=test-table [id=a852d0885c7c44c1806d18fbfc82bf08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:35.524564 2008 log.cc:824] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:35.524605 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d5802eb8b364735846bcccfe2f14a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.525310 2008 tablet_bootstrap.cc:492] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: No bootstrap required, opened a new log
I20260924 19:31:35.525368 2008 ts_tablet_manager.cc:1397] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:31:35.526930 2008 raft_consensus.cc:348] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.527061 2008 raft_consensus.cc:374] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.527081 2008 raft_consensus.cc:729] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0840976c894de6a0bce37c896dc810, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.527180 2008 consensus_queue.cc:260] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.527423 2008 ts_tablet_manager.cc:1428] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:31:35.527441 1708 heartbeater.cc:503] Master 127.0.98.62:42785 was elected leader, sending a full tablet report...
I20260924 19:31:35.527495 2012 tablet_bootstrap.cc:492] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Bootstrap starting.
I20260924 19:31:35.527731 2009 tablet_bootstrap.cc:492] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Bootstrap starting.
I20260924 19:31:35.528082 2012 tablet_bootstrap.cc:654] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.528255 2009 tablet_bootstrap.cc:654] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.528398 2012 log.cc:824] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:35.528514 2009 log.cc:824] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:35.529129 2012 tablet_bootstrap.cc:492] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: No bootstrap required, opened a new log
I20260924 19:31:35.529143 2009 tablet_bootstrap.cc:492] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: No bootstrap required, opened a new log
I20260924 19:31:35.529182 2009 ts_tablet_manager.cc:1397] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:31:35.529184 2012 ts_tablet_manager.cc:1397] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:35.530687 2012 raft_consensus.cc:348] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.530706 2009 raft_consensus.cc:348] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.530815 2009 raft_consensus.cc:374] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.530812 2012 raft_consensus.cc:374] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.530835 2009 raft_consensus.cc:729] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbe26b5345e4e458808231c5e94a6b6, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.530849 2012 raft_consensus.cc:729] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e231a9acaccb4811ba1caf4f7d52bac4, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.530934 2009 consensus_queue.cc:260] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.530975 2012 consensus_queue.cc:260] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.531173 2009 ts_tablet_manager.cc:1428] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:31:35.531189 1848 heartbeater.cc:503] Master 127.0.98.62:42785 was elected leader, sending a full tablet report...
I20260924 19:31:35.531216 2012 ts_tablet_manager.cc:1428] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:31:35.531296 1988 heartbeater.cc:503] Master 127.0.98.62:42785 was elected leader, sending a full tablet report...
W20260924 19:31:35.573823 1849 tablet.cc:2369] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:31:35.677789 1709 tablet.cc:2369] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:31:35.721130 1989 tablet.cc:2369] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:31:35.729627 2015 raft_consensus.cc:482] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:35.729743 2015 raft_consensus.cc:504] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.730094 2015 leader_election.cc:288] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623), e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263)
I20260924 19:31:35.732950 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.730320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44271 (local address 127.0.98.1:43623)
0924 19:31:35.730366 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.730368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.730510 (+ 142us) server_negotiation.cc:407] Connection header received
0924 19:31:35.730541 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.730543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.730555 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.730613 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:35.730756 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.731269 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.732257 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.732606 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.732703 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.732715 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.732843 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.732883 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:31:35.732957 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.730411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43293 (local address 127.0.98.3:40263)
0924 19:31:35.730462 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.730464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.730622 (+ 158us) server_negotiation.cc:407] Connection header received
0924 19:31:35.730657 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.730657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.730667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.730717 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:35.730866 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.731330 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.732255 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.732611 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.732715 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.732725 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.732860 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.732895 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:31:35.732959 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.730292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43623 (local address 127.0.98.2:44271)
0924 19:31:35.730398 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.730409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.730460 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.730623 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.730624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.730625 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:35.730694 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.730697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.731338 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.731338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.732171 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.732174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.732706 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.732706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.732783 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.732786 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.732827 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.732869 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":1}
I20260924 19:31:35.733314 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5802eb8b364735846bcccfe2f14a1f" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0840976c894de6a0bce37c896dc810" is_pre_election: true
I20260924 19:31:35.733485 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.730292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40263 (local address 127.0.98.2:43293)
0924 19:31:35.730523 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.730561 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.730616 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.730732 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.730732 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.730734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:35.730811 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.730812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.731401 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.731401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.732162 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.732164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.732718 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.732719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.732797 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.732800 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.732848 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.732883 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":62,"thread_start_us":98,"threads_started":1}
I20260924 19:31:35.733507 1662 raft_consensus.cc:2454] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbe26b5345e4e458808231c5e94a6b6 in term 0.
I20260924 19:31:35.733816 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5802eb8b364735846bcccfe2f14a1f" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" is_pre_election: true
I20260924 19:31:35.733937 1942 raft_consensus.cc:2454] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbe26b5345e4e458808231c5e94a6b6 in term 0.
I20260924 19:31:35.734012 1733 leader_election.cc:302] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0840976c894de6a0bce37c896dc810, abbe26b5345e4e458808231c5e94a6b6; no voters:
I20260924 19:31:35.734146 2015 raft_consensus.cc:2781] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:35.734202 2015 raft_consensus.cc:482] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:35.734225 2015 raft_consensus.cc:3037] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:35.734925 2015 raft_consensus.cc:504] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.735044 2015 leader_election.cc:288] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623), e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263)
I20260924 19:31:35.735196 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5802eb8b364735846bcccfe2f14a1f" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0840976c894de6a0bce37c896dc810"
I20260924 19:31:35.735214 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d5802eb8b364735846bcccfe2f14a1f" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e231a9acaccb4811ba1caf4f7d52bac4"
I20260924 19:31:35.735291 1942 raft_consensus.cc:3037] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:35.735291 1662 raft_consensus.cc:3037] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:35.736166 1942 raft_consensus.cc:2454] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbe26b5345e4e458808231c5e94a6b6 in term 1.
I20260924 19:31:35.736166 1662 raft_consensus.cc:2454] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbe26b5345e4e458808231c5e94a6b6 in term 1.
I20260924 19:31:35.736330 1733 leader_election.cc:302] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0840976c894de6a0bce37c896dc810, abbe26b5345e4e458808231c5e94a6b6; no voters:
I20260924 19:31:35.736418 2015 raft_consensus.cc:2781] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:35.736550 2015 raft_consensus.cc:686] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 1 LEADER]: Becoming Leader. State: Replica: abbe26b5345e4e458808231c5e94a6b6, State: Running, Role: LEADER
I20260924 19:31:35.736677 2015 consensus_queue.cc:237] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.737432 1505 catalog_manager.cc:5495] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to abbe26b5345e4e458808231c5e94a6b6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20260924 19:31:35.743659 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.741378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.0.98.2:42753)
0924 19:31:35.741418 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.741420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.741446 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:31:35.741537 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.741538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.741556 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.741608 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:35.741723 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.742187 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.742929 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.743111 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.743163 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.743352 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:35.743435 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:35.743503 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.743550 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.743591 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:31:35.745460 1662 raft_consensus.cc:1261] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Refusing update from remote peer abbe26b5345e4e458808231c5e94a6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:35.745460 1942 raft_consensus.cc:1261] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Refusing update from remote peer abbe26b5345e4e458808231c5e94a6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:35.745743 2018 consensus_queue.cc:1035] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:35.745868 2015 consensus_queue.cc:1035] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:35.747890 2021 mvcc.cc:205] Tried to move back new op lower bound from 7332979899370995712 to 7332979899337859072. Current Snapshot: MvccSnapshot[applied={T|T < 7332979899370995712}]
I20260924 19:31:35.747952 2023 mvcc.cc:205] Tried to move back new op lower bound from 7332979899370995712 to 7332979899337859072. Current Snapshot: MvccSnapshot[applied={T|T < 7332979899370995712}]
I20260924 19:31:35.748608 2022 mvcc.cc:205] Tried to move back new op lower bound from 7332979899370995712 to 7332979899337859072. Current Snapshot: MvccSnapshot[applied={T|T < 7332979899370995712}]
I20260924 19:31:35.751670 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:35.751765 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:35.752437 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed: Sending DeleteTablet for 3 replicas of tablet 8d5802eb8b364735846bcccfe2f14a1f
I20260924 19:31:35.752774 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5802eb8b364735846bcccfe2f14a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49676
I20260924 19:31:35.752772 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5802eb8b364735846bcccfe2f14a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36026
I20260924 19:31:35.752779 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d5802eb8b364735846bcccfe2f14a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56880
I20260924 19:31:35.753053 2030 tablet_replica.cc:330] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: stopping tablet replica
I20260924 19:31:35.753181 2030 raft_consensus.cc:2229] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:31:35.753305 2030 raft_consensus.cc:2258] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:35.753327 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:35.753459 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:35.753741 2030 ts_tablet_manager.cc:1905] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:35.754731 1782 tablet_service.cc:1467] Processing CreateTablet for tablet ce851ef88dea48b8aed543372a753af2 (DEFAULT_TABLE table=test-table [id=a6c2c4d6adf44052a28c7280fb2bae09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:35.754822 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce851ef88dea48b8aed543372a753af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.755080 1922 tablet_service.cc:1467] Processing CreateTablet for tablet ce851ef88dea48b8aed543372a753af2 (DEFAULT_TABLE table=test-table [id=a6c2c4d6adf44052a28c7280fb2bae09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:35.755246 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce851ef88dea48b8aed543372a753af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.755293 2032 tablet_replica.cc:330] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: stopping tablet replica
I20260924 19:31:35.755394 2032 raft_consensus.cc:2229] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:35.755492 2032 raft_consensus.cc:2258] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:35.755934 2009 tablet_bootstrap.cc:492] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: Bootstrap starting.
I20260924 19:31:35.756050 2030 ts_tablet_manager.cc:1918] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:35.756109 2030 log.cc:1196] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/8d5802eb8b364735846bcccfe2f14a1f
I20260924 19:31:35.756332 2009 tablet_bootstrap.cc:654] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.756346 2030 ts_tablet_manager.cc:1939] T 8d5802eb8b364735846bcccfe2f14a1f P abbe26b5345e4e458808231c5e94a6b6: Deleting consensus metadata
I20260924 19:31:35.756477 2012 tablet_bootstrap.cc:492] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: Bootstrap starting.
I20260924 19:31:35.756716 2031 tablet_replica.cc:330] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: stopping tablet replica
I20260924 19:31:35.756767 1642 tablet_service.cc:1467] Processing CreateTablet for tablet ce851ef88dea48b8aed543372a753af2 (DEFAULT_TABLE table=test-table [id=a6c2c4d6adf44052a28c7280fb2bae09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:35.756807 2031 raft_consensus.cc:2229] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:35.756860 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce851ef88dea48b8aed543372a753af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.756875 2031 raft_consensus.cc:2258] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:35.756899 2012 tablet_bootstrap.cc:654] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.757225 2009 tablet_bootstrap.cc:492] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: No bootstrap required, opened a new log
I20260924 19:31:35.757253 2009 ts_tablet_manager.cc:1397] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:35.757351 2009 raft_consensus.cc:348] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.757390 2009 raft_consensus.cc:374] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.757401 2009 raft_consensus.cc:729] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbe26b5345e4e458808231c5e94a6b6, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.757436 2009 consensus_queue.cc:260] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.757512 2009 ts_tablet_manager.cc:1428] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.757730 2031 ts_tablet_manager.cc:1905] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:35.758301 2008 tablet_bootstrap.cc:492] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: Bootstrap starting.
I20260924 19:31:35.758553 1491 catalog_manager.cc:4841] TS abbe26b5345e4e458808231c5e94a6b6 (127.0.98.2:42753): tablet 8d5802eb8b364735846bcccfe2f14a1f (table test-table [id=a852d0885c7c44c1806d18fbfc82bf08]) successfully deleted
I20260924 19:31:35.758666 2008 tablet_bootstrap.cc:654] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.758939 2018 raft_consensus.cc:482] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:35.758987 2018 raft_consensus.cc:504] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.759110 2031 ts_tablet_manager.cc:1918] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:35.759138 2018 leader_election.cc:288] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623), e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263)
I20260924 19:31:35.759171 2031 log.cc:1196] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/8d5802eb8b364735846bcccfe2f14a1f
I20260924 19:31:35.759305 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce851ef88dea48b8aed543372a753af2" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" is_pre_election: true
I20260924 19:31:35.759398 2031 ts_tablet_manager.cc:1939] T 8d5802eb8b364735846bcccfe2f14a1f P 6a0840976c894de6a0bce37c896dc810: Deleting consensus metadata
I20260924 19:31:35.759549 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce851ef88dea48b8aed543372a753af2" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0840976c894de6a0bce37c896dc810" is_pre_election: true
W20260924 19:31:35.759577 1736 leader_election.cc:341] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:31:35.759752 1489 catalog_manager.cc:4841] TS 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623): tablet 8d5802eb8b364735846bcccfe2f14a1f (table test-table [id=a852d0885c7c44c1806d18fbfc82bf08]) successfully deleted
W20260924 19:31:35.759775 1733 leader_election.cc:341] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623): Illegal state: must be running to vote when last-logged opid is not known
I20260924 19:31:35.759817 1733 leader_election.cc:302] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abbe26b5345e4e458808231c5e94a6b6; no voters: 6a0840976c894de6a0bce37c896dc810, e231a9acaccb4811ba1caf4f7d52bac4
I20260924 19:31:35.759883 2018 raft_consensus.cc:2726] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:31:35.760133 2032 ts_tablet_manager.cc:1905] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:35.761354 2032 ts_tablet_manager.cc:1918] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:35.761406 2032 log.cc:1196] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/8d5802eb8b364735846bcccfe2f14a1f
I20260924 19:31:35.761595 2032 ts_tablet_manager.cc:1939] T 8d5802eb8b364735846bcccfe2f14a1f P e231a9acaccb4811ba1caf4f7d52bac4: Deleting consensus metadata
I20260924 19:31:35.761937 1492 catalog_manager.cc:4841] TS e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263): tablet 8d5802eb8b364735846bcccfe2f14a1f (table test-table [id=a852d0885c7c44c1806d18fbfc82bf08]) successfully deleted
I20260924 19:31:35.764957 2008 tablet_bootstrap.cc:492] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: No bootstrap required, opened a new log
I20260924 19:31:35.764974 2012 tablet_bootstrap.cc:492] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: No bootstrap required, opened a new log
I20260924 19:31:35.765002 2008 ts_tablet_manager.cc:1397] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260924 19:31:35.765015 2012 ts_tablet_manager.cc:1397] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260924 19:31:35.765154 2012 raft_consensus.cc:348] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.765225 2012 raft_consensus.cc:374] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.765244 2012 raft_consensus.cc:729] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e231a9acaccb4811ba1caf4f7d52bac4, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.765290 2012 consensus_queue.cc:260] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.765311 2008 raft_consensus.cc:348] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.765372 2012 ts_tablet_manager.cc:1428] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:35.765388 2008 raft_consensus.cc:374] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.765413 2008 raft_consensus.cc:729] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0840976c894de6a0bce37c896dc810, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.765452 2008 consensus_queue.cc:260] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.765551 2008 ts_tablet_manager.cc:1428] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:31:35.810820 2016 raft_consensus.cc:482] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:35.810909 2016 raft_consensus.cc:504] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.811187 2016 leader_election.cc:288] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623), abbe26b5345e4e458808231c5e94a6b6 (127.0.98.2:42753)
I20260924 19:31:35.814389 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.811333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43623 (local address 127.0.98.3:45395)
0924 19:31:35.811384 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.811395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.811434 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.811612 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.811613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.811614 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:35.811678 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.811680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.812596 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.812597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.813640 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.813641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.814163 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.814164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.814232 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.814235 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.814276 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.814320 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:31:35.814389 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.811457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45395 (local address 127.0.98.1:43623)
0924 19:31:35.811500 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.811502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.811509 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:31:35.811535 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.811536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.811544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.811588 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:35.811755 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.812515 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.813748 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.814098 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.814156 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.814166 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.814287 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.814325 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:31:35.814642 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.811485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60035 (local address 127.0.98.2:42753)
0924 19:31:35.811546 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.811547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.811606 (+ 59us) server_negotiation.cc:407] Connection header received
0924 19:31:35.811701 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.811702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.811710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.811756 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:35.811866 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.812337 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.813726 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.814263 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.814333 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.814344 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.814532 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.814573 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:31:35.814659 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce851ef88dea48b8aed543372a753af2" candidate_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0840976c894de6a0bce37c896dc810" is_pre_election: true
I20260924 19:31:35.814741 1662 raft_consensus.cc:2454] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e231a9acaccb4811ba1caf4f7d52bac4 in term 0.
I20260924 19:31:35.814898 1873 leader_election.cc:302] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0840976c894de6a0bce37c896dc810, e231a9acaccb4811ba1caf4f7d52bac4; no voters:
I20260924 19:31:35.815048 2016 raft_consensus.cc:2781] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:35.815084 2016 raft_consensus.cc:482] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:35.815109 2016 raft_consensus.cc:3037] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:35.815062 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.811346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42753 (local address 127.0.98.3:60035)
0924 19:31:35.811541 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.811551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.811633 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.811757 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.811758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.811759 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:35.811817 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.811818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.812401 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.812401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.813635 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.813637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.814378 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.814379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.814453 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.814457 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.814749 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.814792 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":62,"threads_started":1}
I20260924 19:31:35.815462 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce851ef88dea48b8aed543372a753af2" candidate_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbe26b5345e4e458808231c5e94a6b6" is_pre_election: true
I20260924 19:31:35.815586 1802 raft_consensus.cc:2454] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e231a9acaccb4811ba1caf4f7d52bac4 in term 0.
I20260924 19:31:35.815917 2016 raft_consensus.cc:504] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.816018 2016 leader_election.cc:288] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623), abbe26b5345e4e458808231c5e94a6b6 (127.0.98.2:42753)
I20260924 19:31:35.816162 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce851ef88dea48b8aed543372a753af2" candidate_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abbe26b5345e4e458808231c5e94a6b6"
I20260924 19:31:35.816179 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce851ef88dea48b8aed543372a753af2" candidate_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0840976c894de6a0bce37c896dc810"
I20260924 19:31:35.816232 1662 raft_consensus.cc:3037] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:35.816232 1802 raft_consensus.cc:3037] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:35.816949 1802 raft_consensus.cc:2454] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e231a9acaccb4811ba1caf4f7d52bac4 in term 1.
I20260924 19:31:35.816949 1662 raft_consensus.cc:2454] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e231a9acaccb4811ba1caf4f7d52bac4 in term 1.
I20260924 19:31:35.817116 1875 leader_election.cc:302] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abbe26b5345e4e458808231c5e94a6b6, e231a9acaccb4811ba1caf4f7d52bac4; no voters:
I20260924 19:31:35.817195 2016 raft_consensus.cc:2781] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:35.817339 2016 raft_consensus.cc:686] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 LEADER]: Becoming Leader. State: Replica: e231a9acaccb4811ba1caf4f7d52bac4, State: Running, Role: LEADER
I20260924 19:31:35.817404 2016 consensus_queue.cc:237] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.818034 1505 catalog_manager.cc:5495] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } health_report { overall_health: HEALTHY } } }
Sep 24 19:31:35 dist-test-slave-m8jb krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:35.859887 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.853264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42785 (local address 127.0.0.1:54830)
0924 19:31:35.853446 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:35.853456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:35.853522 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:35.853682 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:35.853683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:35.853974 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:35.854019 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.854021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.854705 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.854705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.855219 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:35.855220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.855514 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.855515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:35.855569 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.856163 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:35.856168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:35.857528 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:35.858926 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.858933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.858935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.859118 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.859272 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:35.859272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:35.859273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:35.859326 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:35.859528 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:35.859536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:35.859581 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:31:35.859641 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:31:35.859769 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260924 19:31:35.859899 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:35.853393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.0.98.62:42785)
0924 19:31:35.853441 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:35.853443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:35.853517 (+ 74us) server_negotiation.cc:407] Connection header received
0924 19:31:35.853617 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:35.853618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:35.853628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:35.853672 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:35.854095 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.854642 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.855291 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:35.855460 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:35.855511 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:35.857604 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:35.857605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:35.857605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:35.857612 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:35.858857 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.859178 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.859179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.859179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.859223 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:35.859372 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:35.859373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:35.859373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:35.859479 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:35.859531 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:35.859667 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 19:31:35.859825 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":25}
I20260924 19:31:35.861747 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54830:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:31:35.861927 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:35.863214 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 2146951b0e524328a8f038977b9f14a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6af38be94a754c8a9e410a5bd4a5780c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:31:35.863365 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2146951b0e524328a8f038977b9f14a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.864755 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 2146951b0e524328a8f038977b9f14a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6af38be94a754c8a9e410a5bd4a5780c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:31:35.864892 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2146951b0e524328a8f038977b9f14a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.866273 2009 tablet_bootstrap.cc:492] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: Bootstrap starting.
I20260924 19:31:35.866386 1922 tablet_service.cc:1467] Processing CreateTablet for tablet 2146951b0e524328a8f038977b9f14a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6af38be94a754c8a9e410a5bd4a5780c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:31:35.866509 1922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2146951b0e524328a8f038977b9f14a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:35.866613 2009 tablet_bootstrap.cc:654] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.867136 2009 tablet_bootstrap.cc:492] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: No bootstrap required, opened a new log
I20260924 19:31:35.867179 2009 ts_tablet_manager.cc:1397] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:35.867268 2009 raft_consensus.cc:348] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.867334 2009 raft_consensus.cc:374] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.867352 2009 raft_consensus.cc:729] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abbe26b5345e4e458808231c5e94a6b6, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.867393 2009 consensus_queue.cc:260] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.867476 2009 ts_tablet_manager.cc:1428] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.867496 2018 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:35.867568 2018 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:35.867866 2012 tablet_bootstrap.cc:492] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: Bootstrap starting.
I20260924 19:31:35.868243 2012 tablet_bootstrap.cc:654] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.868847 2012 tablet_bootstrap.cc:492] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: No bootstrap required, opened a new log
I20260924 19:31:35.868899 2012 ts_tablet_manager.cc:1397] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:35.869030 2008 tablet_bootstrap.cc:492] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: Bootstrap starting.
I20260924 19:31:35.869026 2012 raft_consensus.cc:348] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.869086 2012 raft_consensus.cc:374] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.869107 2012 raft_consensus.cc:729] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e231a9acaccb4811ba1caf4f7d52bac4, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.869149 2012 consensus_queue.cc:260] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.869241 2012 ts_tablet_manager.cc:1428] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:35.869357 2016 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:35.869426 2016 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:35.869424 2008 tablet_bootstrap.cc:654] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:35.870064 2008 tablet_bootstrap.cc:492] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: No bootstrap required, opened a new log
I20260924 19:31:35.870114 2008 ts_tablet_manager.cc:1397] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:35.870230 2008 raft_consensus.cc:348] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.870283 2008 raft_consensus.cc:374] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:35.870296 2008 raft_consensus.cc:729] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0840976c894de6a0bce37c896dc810, State: Initialized, Role: FOLLOWER
I20260924 19:31:35.870330 2008 consensus_queue.cc:260] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:35.870409 2008 ts_tablet_manager.cc:1428] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:35.870483 2011 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:35.870568 2011 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:36.131356 2018 raft_consensus.cc:482] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:36.131512 2018 raft_consensus.cc:504] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:36.131666 2018 leader_election.cc:288] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623), e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263)
I20260924 19:31:36.131910 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2146951b0e524328a8f038977b9f14a7" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0840976c894de6a0bce37c896dc810" is_pre_election: true
I20260924 19:31:36.131999 1662 raft_consensus.cc:2454] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbe26b5345e4e458808231c5e94a6b6 in term 0.
I20260924 19:31:36.131944 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2146951b0e524328a8f038977b9f14a7" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" is_pre_election: true
I20260924 19:31:36.132053 1942 raft_consensus.cc:2454] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abbe26b5345e4e458808231c5e94a6b6 in term 0.
I20260924 19:31:36.132130 1733 leader_election.cc:302] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a0840976c894de6a0bce37c896dc810, abbe26b5345e4e458808231c5e94a6b6; no voters:
I20260924 19:31:36.132228 2018 raft_consensus.cc:2781] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:36.132261 2018 raft_consensus.cc:482] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:36.132273 2018 raft_consensus.cc:3037] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:36.133118 2018 raft_consensus.cc:504] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:36.133237 2018 leader_election.cc:288] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 election: Requested vote from peers 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623), e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263)
I20260924 19:31:36.133405 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2146951b0e524328a8f038977b9f14a7" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a0840976c894de6a0bce37c896dc810"
I20260924 19:31:36.133424 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2146951b0e524328a8f038977b9f14a7" candidate_uuid: "abbe26b5345e4e458808231c5e94a6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e231a9acaccb4811ba1caf4f7d52bac4"
I20260924 19:31:36.133489 1942 raft_consensus.cc:3037] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:36.133489 1662 raft_consensus.cc:3037] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:36.134182 1662 raft_consensus.cc:2454] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbe26b5345e4e458808231c5e94a6b6 in term 1.
I20260924 19:31:36.134182 1942 raft_consensus.cc:2454] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abbe26b5345e4e458808231c5e94a6b6 in term 1.
I20260924 19:31:36.134371 1736 leader_election.cc:302] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abbe26b5345e4e458808231c5e94a6b6, e231a9acaccb4811ba1caf4f7d52bac4; no voters:
I20260924 19:31:36.134454 2018 raft_consensus.cc:2781] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:36.134501 2018 raft_consensus.cc:686] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [term 1 LEADER]: Becoming Leader. State: Replica: abbe26b5345e4e458808231c5e94a6b6, State: Running, Role: LEADER
I20260924 19:31:36.134550 2018 consensus_queue.cc:237] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } }
I20260924 19:31:36.134805 2015 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: TxnStatusTablet state changed. Reason: New leader abbe26b5345e4e458808231c5e94a6b6. Latest consensus state: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:36.134869 2015 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:31:36.134989 2049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:31:36.135113 2049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:31:36.135138 1505 catalog_manager.cc:5495] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to abbe26b5345e4e458808231c5e94a6b6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } health_report { overall_health: UNKNOWN } } }
I20260924 19:31:36.148744 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.146225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.0.98.2:42753)
0924 19:31:36.146261 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:36.146263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:36.146302 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:31:36.146387 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:36.146388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:36.146398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:36.146443 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:36.146561 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.147013 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.148089 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.148274 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.148328 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.148498 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:36.148523 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:36.148552 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:36.148621 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:31:36.148679 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:31:36.148772 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.146179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42753 (local address 127.0.0.1:36044)
0924 19:31:36.146261 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:36.146272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:36.146321 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:36.146455 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:36.146456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:36.146457 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:36.146509 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:36.146512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.147088 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.147089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:36.147998 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:36.148001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.148336 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.148336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:36.148401 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.148418 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:36.148565 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:36.148571 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:31:36.148618 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:31:36.148689 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 19:31:36.149686 1942 raft_consensus.cc:1261] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Refusing update from remote peer abbe26b5345e4e458808231c5e94a6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:36.149704 1662 raft_consensus.cc:1261] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Refusing update from remote peer abbe26b5345e4e458808231c5e94a6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:36.149899 2018 consensus_queue.cc:1035] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:36.149996 2015 consensus_queue.cc:1035] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:36.150542 2016 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: TxnStatusTablet state changed. Reason: New leader abbe26b5345e4e458808231c5e94a6b6. Latest consensus state: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:36.150645 2016 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:36.150701 2011 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: TxnStatusTablet state changed. Reason: New leader abbe26b5345e4e458808231c5e94a6b6. Latest consensus state: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:36.150780 2011 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:36.151005 2018 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:36.151109 2018 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:31:36.151739 2011 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:36.151823 2011 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P 6a0840976c894de6a0bce37c896dc810: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:36.152537 2018 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:36.152609 2018 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P abbe26b5345e4e458808231c5e94a6b6: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:31:36.152817 2016 tablet_replica.cc:439] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abbe26b5345e4e458808231c5e94a6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } } peers { permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 } } peers { permanent_uuid: "e231a9acaccb4811ba1caf4f7d52bac4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40263 } } }
I20260924 19:31:36.152884 2016 tablet_replica.cc:442] T 2146951b0e524328a8f038977b9f14a7 P e231a9acaccb4811ba1caf4f7d52bac4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:36.155434 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.152551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.98.3:40263)
0924 19:31:36.152593 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:36.152594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:36.152701 (+ 107us) server_negotiation.cc:407] Connection header received
0924 19:31:36.152737 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:36.152737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:36.152757 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:36.152813 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:36.153067 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.153890 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.154622 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.154935 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.155003 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.155128 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:36.155255 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:36.155327 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:36.155341 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 19:31:36.155360 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:31:36.160072 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.157852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36905 (local address 127.0.98.2:42753)
0924 19:31:36.157904 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:36.157905 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:36.158020 (+ 115us) server_negotiation.cc:407] Connection header received
0924 19:31:36.158078 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:36.158079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:36.158090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:36.158133 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:36.158269 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.158761 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.159460 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.159656 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.159710 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.159866 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:36.159896 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:36.159934 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:36.159982 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 19:31:36.160002 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:31:36.160286 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.157803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42753 (local address 127.0.98.3:36905)
0924 19:31:36.157966 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:36.157974 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:36.158023 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:36.158161 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:36.158161 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:36.158163 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:36.158218 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:36.158219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.158829 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.158830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:36.159392 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:36.159393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.159734 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.159735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:36.159792 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.159804 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:36.159940 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:36.159942 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:31:36.159981 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 19:31:36.160019 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260924 19:31:36.162209 1662 raft_consensus.cc:1261] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Refusing update from remote peer e231a9acaccb4811ba1caf4f7d52bac4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:36.162341 1802 raft_consensus.cc:1261] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Refusing update from remote peer e231a9acaccb4811ba1caf4f7d52bac4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:36.162425 2016 consensus_queue.cc:1035] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a0840976c894de6a0bce37c896dc810" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:36.162639 2016 consensus_queue.cc:1035] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abbe26b5345e4e458808231c5e94a6b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:36.170302 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.168114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42135 (local address 127.0.98.3:40263)
0924 19:31:36.168158 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:36.168160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:36.168182 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:31:36.168200 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:36.168201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:36.168211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:36.168249 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:36.168334 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.168834 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.169563 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.169861 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.169929 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.170045 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:36.170077 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:36.170120 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:36.170217 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 19:31:36.170240 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:31:36.170328 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.167965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40263 (local address 127.0.98.2:42135)
0924 19:31:36.168100 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:36.168111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:36.168155 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:36.168250 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:36.168251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:36.168252 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:36.168293 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:36.168294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.168902 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.168903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:36.169491 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:36.169493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.169926 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.169927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:36.169968 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.169981 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:36.170142 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:36.170144 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:31:36.170205 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:31:36.170251 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
I20260924 19:31:36.176514 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:36.176617 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:36.177214 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54a1e25f156a49549be1521fabdc6bed: Sending DeleteTablet for 3 replicas of tablet ce851ef88dea48b8aed543372a753af2
I20260924 19:31:36.177451 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet ce851ef88dea48b8aed543372a753af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56880
I20260924 19:31:36.177542 2031 tablet_replica.cc:330] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: stopping tablet replica
I20260924 19:31:36.177585 2031 raft_consensus.cc:2229] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:36.177618 2031 raft_consensus.cc:2258] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:36.177829 1922 tablet_service.cc:1514] Processing DeleteTablet for tablet ce851ef88dea48b8aed543372a753af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49676
I20260924 19:31:36.177911 2032 tablet_replica.cc:330] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: stopping tablet replica
I20260924 19:31:36.177954 2032 raft_consensus.cc:2229] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:31:36.178032 2032 raft_consensus.cc:2258] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:36.178347 2032 ts_tablet_manager.cc:1905] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:36.178535 2031 ts_tablet_manager.cc:1905] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:36.180053 2032 ts_tablet_manager.cc:1918] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:36.180107 2032 log.cc:1196] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/ce851ef88dea48b8aed543372a753af2
I20260924 19:31:36.180096 2031 ts_tablet_manager.cc:1918] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:36.180150 2031 log.cc:1196] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/ce851ef88dea48b8aed543372a753af2
I20260924 19:31:36.180305 2032 ts_tablet_manager.cc:1939] T ce851ef88dea48b8aed543372a753af2 P e231a9acaccb4811ba1caf4f7d52bac4: Deleting consensus metadata
I20260924 19:31:36.180337 2031 ts_tablet_manager.cc:1939] T ce851ef88dea48b8aed543372a753af2 P 6a0840976c894de6a0bce37c896dc810: Deleting consensus metadata
I20260924 19:31:36.180436 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet ce851ef88dea48b8aed543372a753af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36026
I20260924 19:31:36.180680 2030 tablet_replica.cc:330] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: stopping tablet replica
I20260924 19:31:36.180747 2030 raft_consensus.cc:2229] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:36.180788 2030 raft_consensus.cc:2258] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:36.180763 1489 catalog_manager.cc:4841] TS 6a0840976c894de6a0bce37c896dc810 (127.0.98.1:43623): tablet ce851ef88dea48b8aed543372a753af2 (table test-table [id=a6c2c4d6adf44052a28c7280fb2bae09]) successfully deleted
I20260924 19:31:36.181123 1492 catalog_manager.cc:4841] TS e231a9acaccb4811ba1caf4f7d52bac4 (127.0.98.3:40263): tablet ce851ef88dea48b8aed543372a753af2 (table test-table [id=a6c2c4d6adf44052a28c7280fb2bae09]) successfully deleted
I20260924 19:31:36.181198 2030 ts_tablet_manager.cc:1905] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:36.182618 2030 ts_tablet_manager.cc:1918] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:36.182682 2030 log.cc:1196] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/ce851ef88dea48b8aed543372a753af2
I20260924 19:31:36.182898 2030 ts_tablet_manager.cc:1939] T ce851ef88dea48b8aed543372a753af2 P abbe26b5345e4e458808231c5e94a6b6: Deleting consensus metadata
I20260924 19:31:36.183223 1491 catalog_manager.cc:4841] TS abbe26b5345e4e458808231c5e94a6b6 (127.0.98.2:42753): tablet ce851ef88dea48b8aed543372a753af2 (table test-table [id=a6c2c4d6adf44052a28c7280fb2bae09]) successfully deleted
Sep 24 19:31:36 dist-test-slave-m8jb krb5kdc[1460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:36.191044 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.180861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56884 (local address 127.0.98.1:43623)
0924 19:31:36.180902 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:36.180904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:36.181284 (+ 380us) server_negotiation.cc:407] Connection header received
0924 19:31:36.181375 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:36.181376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:36.181388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:36.181415 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:36.182168 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.182899 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.184036 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.184285 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.184324 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.187721 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:36.187731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:36.187731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:36.187739 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:36.189775 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:36.190167 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:36.190171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:36.190171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:36.190236 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:36.190425 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:36.190426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:36.190426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:36.190598 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:36.190747 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:36.190842 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 19:31:36.190926 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:31:36.191293 1702 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56884
I20260924 19:31:36.199268 1573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:36.192163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.0.98.62:42785)
0924 19:31:36.192205 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:36.192207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:36.192420 (+ 213us) server_negotiation.cc:407] Connection header received
0924 19:31:36.192493 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:36.192494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:36.192507 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:36.192551 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:36.193139 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.193971 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.194987 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:36.195311 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:36.195380 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:36.196589 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:36.196590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:36.196590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:36.196598 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:36.198181 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:36.198542 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:36.198543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:36.198543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:36.198601 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:36.198772 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:36.198773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:36.198773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:36.198899 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:36.198996 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:36.199089 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 19:31:36.199172 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 19:31:36.199453 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54844
I20260924 19:31:36.200136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 1577
I20260924 19:31:36.207350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 1717
I20260924 19:31:36.213068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 1857
I20260924 19:31:36.217612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 1473
2026-09-24T19:31:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2401 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:36 dist-test-slave-m8jb krb5kdc[2070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:36 dist-test-slave-m8jb krb5kdc[2070](info): set up 2 sockets
Sep 24 19:31:36 dist-test-slave-m8jb krb5kdc[2070](info): commencing operation
krb5kdc: starting...
W20260924 19:31:37.156173 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:37.267810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43715
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:37.345716 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:37.345930 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:37.345952 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:37.347097 2083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:37.347136 2083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:31:37.347148 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:37.347160 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:37.347170 2083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:37.347182 2083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:37.348621 2083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:37.348865 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:37.349092 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:37.353791 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:37.353825 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:37.353945 2083 server_base.cc:1034] running on GCE node
W20260924 19:31:37.353791 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:37.355043 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:37.355317 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:37.356458 2083 hybrid_clock.cc:648] HybridClock initialized: now 1790278297356439 us; error 37 us; skew 500 ppm
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:37.358956 2083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:37.359349 2083 webserver.cc:460] Webserver started at http://127.0.98.62:39185/ using document root <none> and password file <none>
I20260924 19:31:37.359531 2083 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:37.359570 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:37.359659 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:37.360654 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "a14dcbbbd16c48f5ae3d06badd258729"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "d64c37ff7c9a00ff554be754118c2687"
server_key_iv: "50f986cc80c9fb6aa1654e6d17655b6a"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.360970 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "a14dcbbbd16c48f5ae3d06badd258729"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "d64c37ff7c9a00ff554be754118c2687"
server_key_iv: "50f986cc80c9fb6aa1654e6d17655b6a"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.362113 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:37.362792 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.362927 2083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.363004 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "a14dcbbbd16c48f5ae3d06badd258729"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "d64c37ff7c9a00ff554be754118c2687"
server_key_iv: "50f986cc80c9fb6aa1654e6d17655b6a"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.363068 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:37.377374 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:37.379248 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:37.379369 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:37.383847 2083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43715
I20260924 19:31:37.383898 2160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43715 every 8 connection(s)
I20260924 19:31:37.384219 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:37.384917 2161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:37.387089 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: Bootstrap starting.
I20260924 19:31:37.387633 2161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:37.387872 2161 log.cc:824] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:37.388480 2161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: No bootstrap required, opened a new log
I20260924 19:31:37.390512 2161 raft_consensus.cc:348] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14dcbbbd16c48f5ae3d06badd258729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43715 } }
I20260924 19:31:37.390657 2161 raft_consensus.cc:374] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:37.390689 2161 raft_consensus.cc:729] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14dcbbbd16c48f5ae3d06badd258729, State: Initialized, Role: FOLLOWER
I20260924 19:31:37.390820 2161 consensus_queue.cc:260] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14dcbbbd16c48f5ae3d06badd258729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43715 } }
I20260924 19:31:37.390880 2161 raft_consensus.cc:388] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:37.390909 2161 raft_consensus.cc:482] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:37.390952 2161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:37.391427 2161 raft_consensus.cc:504] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14dcbbbd16c48f5ae3d06badd258729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43715 } }
I20260924 19:31:37.391516 2161 leader_election.cc:302] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a14dcbbbd16c48f5ae3d06badd258729; no voters:
I20260924 19:31:37.391664 2161 leader_election.cc:288] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:37.391701 2164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:37.391849 2164 raft_consensus.cc:686] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [term 1 LEADER]: Becoming Leader. State: Replica: a14dcbbbd16c48f5ae3d06badd258729, State: Running, Role: LEADER
I20260924 19:31:37.391891 2161 sys_catalog.cc:564] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:37.391937 2164 consensus_queue.cc:237] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14dcbbbd16c48f5ae3d06badd258729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43715 } }
I20260924 19:31:37.392267 2165 sys_catalog.cc:455] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a14dcbbbd16c48f5ae3d06badd258729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14dcbbbd16c48f5ae3d06badd258729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43715 } } }
I20260924 19:31:37.392352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 2083
I20260924 19:31:37.392364 2165 sys_catalog.cc:458] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:37.392442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:37.392287 2166 sys_catalog.cc:455] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a14dcbbbd16c48f5ae3d06badd258729. Latest consensus state: current_term: 1 leader_uuid: "a14dcbbbd16c48f5ae3d06badd258729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14dcbbbd16c48f5ae3d06badd258729" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43715 } } }
I20260924 19:31:37.392555 2166 sys_catalog.cc:458] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:37.392616 392 external_mini_cluster.cc:1428] Setting key fc661dd556b02ad57f61cd7e3ba60cad
I20260924 19:31:37.393071 2172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:37.393620 2172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:37.394953 2172 catalog_manager.cc:1347] Generated new cluster ID: bb74ec83b088448c8bc30186eeed4717
I20260924 19:31:37.395001 2172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:37.403944 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.393053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.0.98.62:43715)
0924 19:31:37.393443 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.393453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.393486 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:31:37.393611 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.393618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.393677 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.393820 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:37.394483 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.395830 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.396932 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.397504 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.397608 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.400247 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:37.400251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:37.400252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:37.400265 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:37.402240 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.402602 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.402605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.402606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.402735 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.402906 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.402906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.402907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.403231 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:37.403380 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.403447 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.403695 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":75,"threads_started":1}
I20260924 19:31:37.417301 2172 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:37.417814 2172 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:37.421460 2172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: Generated new TSK 0
I20260924 19:31:37.421617 2172 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:37.453006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43715
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:37.545645 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:37.545861 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:37.545885 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:37.547015 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:37.547063 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:37.547086 2187 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:31:37.547108 2187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:31:37.547137 2187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:37.548592 2187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:37.548837 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:37.549041 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:31:37.553864 2187 server_base.cc:1034] running on GCE node
W20260924 19:31:37.553841 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:37.553851 2193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:37.553849 2195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:37.554250 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:37.554479 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:37.555612 2187 hybrid_clock.cc:648] HybridClock initialized: now 1790278297555589 us; error 41 us; skew 500 ppm
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:37.558033 2187 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:37.558439 2187 webserver.cc:460] Webserver started at http://127.0.98.1:34965/ using document root <none> and password file <none>
I20260924 19:31:37.558640 2187 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:37.558691 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:37.558812 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:37.559835 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "d909237adf8645a190f96bbb1f9fcceb"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "739b35175d0636f1104e8886a1654971"
server_key_iv: "e8ec59dc69ca487330377143c0216d01"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.560158 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d909237adf8645a190f96bbb1f9fcceb"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "739b35175d0636f1104e8886a1654971"
server_key_iv: "e8ec59dc69ca487330377143c0216d01"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.561405 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:37.562062 2202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.562243 2187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:37.562322 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "d909237adf8645a190f96bbb1f9fcceb"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "739b35175d0636f1104e8886a1654971"
server_key_iv: "e8ec59dc69ca487330377143c0216d01"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.562376 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:37.589111 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:37.591310 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:37.591419 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:37.591935 2187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:37.592001 2187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.592024 2187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:37.592036 2187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.598265 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45389
I20260924 19:31:37.598348 2317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45389 every 8 connection(s)
I20260924 19:31:37.598666 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:37.601671 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.592319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38581 (local address 127.0.98.62:43715)
0924 19:31:37.592386 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.592389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.592683 (+ 294us) server_negotiation.cc:407] Connection header received
0924 19:31:37.593257 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.593259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.593272 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.593351 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:37.594140 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.594905 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.596392 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.596740 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.596821 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.599233 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:37.599234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:37.599234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:37.599242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:37.600535 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.600903 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.600904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.600905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.600965 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.601180 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.601181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.601181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.601282 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:37.601327 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.601538 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.601607 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:31:37.601838 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.592336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43715 (local address 127.0.98.1:38581)
0924 19:31:37.592522 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.592564 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.593136 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.593410 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.593413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.593752 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:37.594036 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.594042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.594999 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.595000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.596297 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.596299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.596820 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.596821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.596930 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.597430 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:37.597442 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:37.599132 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:37.600601 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.600606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.600608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.600838 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.601019 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.601019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.601019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.601119 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.601324 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:37.601328 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:37.601440 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.601513 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.601632 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260924 19:31:37.602295 2318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43715
I20260924 19:31:37.602380 2318 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:37.602557 2318 heartbeater.cc:511] Master 127.0.98.62:43715 requested a full tablet report, sending...
I20260924 19:31:37.603119 2114 ts_manager.cc:194] Registered new tserver with Master: d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389)
I20260924 19:31:37.603838 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38581
I20260924 19:31:37.608433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 2187
I20260924 19:31:37.608534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:37.608732 392 external_mini_cluster.cc:1428] Setting key 59b11f3d772c1cdb3a64a2ac8b4f635b
I20260924 19:31:37.608960 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.603263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43715 (local address 127.0.98.1:37495)
0924 19:31:37.603404 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.603415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.603481 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.604169 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.604170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.604312 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:37.604362 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.604363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.605029 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.605030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.605523 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.605525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.605840 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.605841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.605907 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.606232 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:37.606236 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:37.606484 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:37.607920 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.607921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.607921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.608089 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.608268 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.608269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.608269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.608324 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.608618 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:37.608619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:37.608688 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.608756 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.608857 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260924 19:31:37.609066 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.604010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37495 (local address 127.0.98.62:43715)
0924 19:31:37.604045 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.604046 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.604054 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:37.604071 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.604072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.604080 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.604132 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:37.604426 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.604942 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.605588 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.605784 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.605836 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.606558 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:37.606559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:37.606559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:37.606566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:37.607856 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.608146 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.608147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.608148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.608219 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.608398 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.608399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.608400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.608536 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:37.608591 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.608853 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.608975 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:37.633548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43715
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:37.711534 2327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:37.711747 2327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:37.711781 2327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:37.712929 2327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:37.712966 2327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:37.712993 2327 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:31:37.713007 2327 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:31:37.713029 2327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:37.714434 2327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:37.714673 2327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:37.714879 2327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:37.719558 2335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:37.719534 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:37.719553 2333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:37.719763 2327 server_base.cc:1034] running on GCE node
I20260924 19:31:37.719980 2327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:37.720220 2327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:37.721355 2327 hybrid_clock.cc:648] HybridClock initialized: now 1790278297721341 us; error 25 us; skew 500 ppm
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:37.723577 2327 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:37.724000 2327 webserver.cc:460] Webserver started at http://127.0.98.2:43801/ using document root <none> and password file <none>
I20260924 19:31:37.724197 2327 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:37.724243 2327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:37.724368 2327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:37.725433 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "4fe786d7395b4d439c1484587c4f0448"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "0e61702d019717177a8641104fec6e47"
server_key_iv: "1e9a94e7f8b2c9d3c6a772cf063f733a"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.725780 2327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4fe786d7395b4d439c1484587c4f0448"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "0e61702d019717177a8641104fec6e47"
server_key_iv: "1e9a94e7f8b2c9d3c6a772cf063f733a"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.726977 2327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:37.727602 2342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.727749 2327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:37.727829 2327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "4fe786d7395b4d439c1484587c4f0448"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "0e61702d019717177a8641104fec6e47"
server_key_iv: "1e9a94e7f8b2c9d3c6a772cf063f733a"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.727886 2327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:37.746018 2327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:37.748070 2327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:37.748215 2327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:37.748814 2327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:37.748863 2327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.748893 2327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:37.748909 2327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:37.755883 2327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42363
I20260924 19:31:37.755944 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42363 every 8 connection(s)
I20260924 19:31:37.756292 2327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:31:37.757763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 2327
I20260924 19:31:37.757854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:37.758023 392 external_mini_cluster.cc:1428] Setting key 244b5a072bbd3d3d50ac6b3a65c6446d
I20260924 19:31:37.758700 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.749288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46555 (local address 127.0.98.62:43715)
0924 19:31:37.749358 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.749360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.749593 (+ 233us) server_negotiation.cc:407] Connection header received
0924 19:31:37.750286 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.750288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.750304 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.750374 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:37.751219 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.751750 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.753249 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.753436 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.753495 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.755943 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:37.755944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:37.755945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:37.755953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:37.757310 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.757713 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.757714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.757715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.757778 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.758013 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.758014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.758015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.758152 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:37.758233 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.758517 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.758590 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:31:37.758937 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.749262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43715 (local address 127.0.98.2:46555)
0924 19:31:37.749458 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.749509 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.750162 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.750467 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.750471 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.750831 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:37.751129 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.751135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.751837 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.751838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.753160 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.753163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.753525 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.753526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.753686 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.754201 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:37.754211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:37.755829 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:37.757412 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.757418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.757420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.757643 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.757841 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.757841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.757842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.757953 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.758231 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:37.758238 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:37.758394 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.758499 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.758717 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260924 19:31:37.759331 2458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43715
I20260924 19:31:37.759411 2458 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:37.759579 2458 heartbeater.cc:511] Master 127.0.98.62:43715 requested a full tablet report, sending...
I20260924 19:31:37.759930 2114 ts_manager.cc:194] Registered new tserver with Master: 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363)
I20260924 19:31:37.760388 2114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46555
I20260924 19:31:37.765889 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.760297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43715 (local address 127.0.98.2:54301)
0924 19:31:37.760421 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.760430 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.760499 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.760609 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.760610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.760812 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:37.760873 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.760874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.761486 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.761487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.761995 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.761996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.762383 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.762383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.762456 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.762831 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:37.762835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:37.763140 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:37.764774 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.764775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.764775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.764979 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.765168 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.765169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.765169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.765216 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.765549 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:37.765550 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:37.765626 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.765701 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.765810 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260924 19:31:37.765901 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.760351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54301 (local address 127.0.98.62:43715)
0924 19:31:37.760385 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.760387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.760487 (+ 100us) server_negotiation.cc:407] Connection header received
0924 19:31:37.760543 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.760543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.760553 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.760596 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:37.760942 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.761399 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.762065 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.762312 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.762373 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.763231 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:37.763233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:37.763233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:37.763239 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:37.764704 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.765049 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.765050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.765051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.765108 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.765272 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.765273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.765274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.765467 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:37.765534 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.765735 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.765812 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:37.783670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43715
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:37.862231 2467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:37.862426 2467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:37.862458 2467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:37.863602 2467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:37.863643 2467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:37.863669 2467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:31:37.863684 2467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:31:37.863708 2467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:37.865211 2467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:37.865433 2467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:37.865653 2467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:37.870512 2473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:37.870532 2472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:37.870617 2467 server_base.cc:1034] running on GCE node
W20260924 19:31:37.870520 2475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:37.870900 2467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:37.871145 2467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:37.872300 2467 hybrid_clock.cc:648] HybridClock initialized: now 1790278297872287 us; error 25 us; skew 500 ppm
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:37.874706 2467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:37.875126 2467 webserver.cc:460] Webserver started at http://127.0.98.3:36657/ using document root <none> and password file <none>
I20260924 19:31:37.875321 2467 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:37.875368 2467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:37.875494 2467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:37.876506 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "edb80b2c0f1e41608a9437427f8f08eb"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "1dcadb3b7438e289a55117505ac4ddb0"
server_key_iv: "3ba97bd353723b340cf6711127beca37"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.876863 2467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "edb80b2c0f1e41608a9437427f8f08eb"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "1dcadb3b7438e289a55117505ac4ddb0"
server_key_iv: "3ba97bd353723b340cf6711127beca37"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.878199 2467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:37.878895 2482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.879043 2467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:37.879130 2467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "edb80b2c0f1e41608a9437427f8f08eb"
format_stamp: "Formatted at 2026-09-24 19:31:37 on dist-test-slave-m8jb"
server_key: "1dcadb3b7438e289a55117505ac4ddb0"
server_key_iv: "3ba97bd353723b340cf6711127beca37"
server_key_version: "encryptionkey@0"
I20260924 19:31:37.879186 2467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:37.894369 2467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:37.896483 2467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:37.896617 2467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:37.897182 2467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:37.897222 2467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.897250 2467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:37.897267 2467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:37.903291 2467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39133
I20260924 19:31:37.903368 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39133 every 8 connection(s)
I20260924 19:31:37.903709 2467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:37.906445 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.897617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56177 (local address 127.0.98.62:43715)
0924 19:31:37.897674 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.897676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.897908 (+ 232us) server_negotiation.cc:407] Connection header received
0924 19:31:37.898430 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.898432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.898446 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.898520 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:37.899301 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.900043 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.901551 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.901790 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.901849 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.904264 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:37.904265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:37.904265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:37.904272 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:37.905427 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.905792 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.905793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.905793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.905837 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.905992 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.905992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.905993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.906085 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:37.906127 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.906329 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.906383 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:31:37.906659 2491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.897628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43715 (local address 127.0.98.3:56177)
0924 19:31:37.897788 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.897824 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.898333 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.898577 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.898582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.898913 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:37.899203 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.899210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.900163 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.900164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.901473 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.901477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.901859 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.901860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.901997 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.902550 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:37.902560 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:37.904181 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:37.905491 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.905496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.905498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.905734 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.905877 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.905877 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.905878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.905960 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.906126 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:37.906129 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:37.906240 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.906315 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.906427 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260924 19:31:37.907087 2598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43715
I20260924 19:31:37.907166 2598 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:37.907330 2598 heartbeater.cc:511] Master 127.0.98.62:43715 requested a full tablet report, sending...
I20260924 19:31:37.907658 2115 ts_manager.cc:194] Registered new tserver with Master: edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:37.908047 2115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56177
I20260924 19:31:37.908165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 2467
I20260924 19:31:37.908248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:37.908425 392 external_mini_cluster.cc:1428] Setting key 37e0f1115e12c8a38f7b3d7a70eef79a
I20260924 19:31:37.909186 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:31:37.913817 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.908037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54267 (local address 127.0.98.62:43715)
0924 19:31:37.908113 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.908115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.908660 (+ 545us) server_negotiation.cc:407] Connection header received
0924 19:31:37.908721 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.908722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.908731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.908774 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:37.909104 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.909626 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.910532 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.910714 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.910772 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.911630 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:37.911632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:37.911632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:37.911639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:37.912880 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.913164 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.913165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.913165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.913216 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.913366 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.913367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.913367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.913467 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:37.913519 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.913652 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.913721 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 19:31:37.913829 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.907991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43715 (local address 127.0.98.3:54267)
0924 19:31:37.908570 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.908581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.908663 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.908777 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.908777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.908972 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:37.909030 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.909032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.909717 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.909718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.910447 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.910450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.910784 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.910785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.910861 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.911249 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:37.911253 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:37.911524 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:37.912949 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.912950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.912951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.913106 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.913264 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:37.913265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:37.913265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:37.913318 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:37.913516 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:37.913517 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:37.913565 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.913624 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.913722 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":68,"threads_started":1}
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:31:37 dist-test-slave-m8jb krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:37.931291 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.924263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.0.98.62:43715)
0924 19:31:37.924319 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.924324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.924368 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:31:37.924429 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.924430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.924439 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.924487 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:37.925053 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.925539 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.926329 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.926516 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.926623 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.929146 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:37.929148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:37.929148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:37.929156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:37.930406 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.930729 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.930731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.930731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.930784 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:37.930919 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:37.930920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:37.930920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:37.931018 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:37.931089 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.931169 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.931226 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:31:37.932765 2115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:37.934242 2115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:37.940171 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.936318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42363 (local address 127.0.0.1:44496)
0924 19:31:37.936473 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.936516 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.936581 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.936877 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.936880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.936882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:37.936948 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.936949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.938336 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.938337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.939139 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.939142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.939872 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.939873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.939956 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.939970 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.940038 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.940085 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":30,"thread_start_us":47,"threads_started":1}
I20260924 19:31:37.940196 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.936501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.0.98.2:42363)
0924 19:31:37.936742 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.936748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.936756 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:37.936786 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.936787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.936800 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.936873 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:37.937031 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.938225 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.939221 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.939798 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.939870 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.939928 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.940063 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.940100 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":103,"threads_started":1}
I20260924 19:31:37.940716 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.936342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39133 (local address 127.0.0.1:39026)
0924 19:31:37.936727 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.936738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.936788 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.937087 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.937088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.937091 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:37.937139 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.937141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.938337 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.938338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.939657 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.939658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.940488 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.940489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.940547 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.940551 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.940596 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.940652 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260924 19:31:37.940762 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.936438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.0.98.3:39133)
0924 19:31:37.936893 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.936897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.936906 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:37.936941 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.936942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.936953 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.937013 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:37.937209 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.938268 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.939721 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.940414 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.940479 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.940534 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.940617 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.940678 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":128,"threads_started":1}
I20260924 19:31:37.940909 2392 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc872c6fb646f6a8062b4da323b483 (DEFAULT_TABLE table=test-table [id=455f8f0d722045cda18e06a82616442d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:37.941133 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.936497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.0.1:59850)
0924 19:31:37.936948 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:37.936958 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:37.936995 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:37.937323 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:37.937324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:37.937326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:37.937361 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.937363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.938182 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.938183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.939430 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:37.939432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.940672 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.940673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:37.940732 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.940736 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:37.941026 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 19:31:37.941070 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":33,"thread_start_us":60,"threads_started":1}
I20260924 19:31:37.941300 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc872c6fb646f6a8062b4da323b483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:37.941370 2532 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc872c6fb646f6a8062b4da323b483 (DEFAULT_TABLE table=test-table [id=455f8f0d722045cda18e06a82616442d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:37.941480 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:37.936738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.0.98.1:45389)
0924 19:31:37.937131 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:37.937136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:37.937144 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:37.937177 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:37.937178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:37.937192 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:37.937259 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:37.937413 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.938090 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.939504 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:37.940203 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:37.940604 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:37.941313 (+ 709us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:37.941339 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 19:31:37.941379 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":80,"threads_started":1}
I20260924 19:31:37.941696 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc872c6fb646f6a8062b4da323b483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:37.942052 2252 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc872c6fb646f6a8062b4da323b483 (DEFAULT_TABLE table=test-table [id=455f8f0d722045cda18e06a82616442d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:37.942343 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc872c6fb646f6a8062b4da323b483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:37.944044 2621 tablet_bootstrap.cc:492] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Bootstrap starting.
I20260924 19:31:37.944576 2621 tablet_bootstrap.cc:654] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:37.944737 2622 tablet_bootstrap.cc:492] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Bootstrap starting.
I20260924 19:31:37.944911 2621 log.cc:824] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:37.945562 2622 tablet_bootstrap.cc:654] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:37.945587 2624 tablet_bootstrap.cc:492] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Bootstrap starting.
I20260924 19:31:37.945865 2621 tablet_bootstrap.cc:492] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: No bootstrap required, opened a new log
I20260924 19:31:37.945865 2622 log.cc:824] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:37.945930 2621 ts_tablet_manager.cc:1397] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:37.946167 2624 tablet_bootstrap.cc:654] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:37.946480 2624 log.cc:824] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:37.946585 2622 tablet_bootstrap.cc:492] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: No bootstrap required, opened a new log
I20260924 19:31:37.946662 2622 ts_tablet_manager.cc:1397] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:37.947232 2624 tablet_bootstrap.cc:492] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: No bootstrap required, opened a new log
I20260924 19:31:37.947297 2624 ts_tablet_manager.cc:1397] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:37.947554 2621 raft_consensus.cc:348] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:37.947695 2621 raft_consensus.cc:374] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:37.947724 2621 raft_consensus.cc:729] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe786d7395b4d439c1484587c4f0448, State: Initialized, Role: FOLLOWER
I20260924 19:31:37.947854 2621 consensus_queue.cc:260] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:37.948118 2458 heartbeater.cc:503] Master 127.0.98.62:43715 was elected leader, sending a full tablet report...
I20260924 19:31:37.948107 2621 ts_tablet_manager.cc:1428] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:31:37.948485 2622 raft_consensus.cc:348] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:37.948673 2622 raft_consensus.cc:374] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:37.948714 2622 raft_consensus.cc:729] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb80b2c0f1e41608a9437427f8f08eb, State: Initialized, Role: FOLLOWER
I20260924 19:31:37.948823 2622 consensus_queue.cc:260] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:37.949067 2622 ts_tablet_manager.cc:1428] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:31:37.949164 2598 heartbeater.cc:503] Master 127.0.98.62:43715 was elected leader, sending a full tablet report...
I20260924 19:31:37.949671 2624 raft_consensus.cc:348] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:37.949820 2624 raft_consensus.cc:374] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:37.949864 2624 raft_consensus.cc:729] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d909237adf8645a190f96bbb1f9fcceb, State: Initialized, Role: FOLLOWER
I20260924 19:31:37.949970 2624 consensus_queue.cc:260] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:37.950194 2624 ts_tablet_manager.cc:1428] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:31:37.950240 2318 heartbeater.cc:503] Master 127.0.98.62:43715 was elected leader, sending a full tablet report...
W20260924 19:31:38.007349 2459 tablet.cc:2369] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:31:38.088675 2629 raft_consensus.cc:482] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:38.088798 2629 raft_consensus.cc:504] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.089179 2629 leader_election.cc:288] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363), edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:38.091929 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.089408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33371 (local address 127.0.98.2:42363)
0924 19:31:38.089449 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.089451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.089591 (+ 140us) server_negotiation.cc:407] Connection header received
0924 19:31:38.089623 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.089624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.089635 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.089702 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:38.089900 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.090343 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.091237 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.091587 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.091640 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.091651 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.091813 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.091867 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:31:38.091929 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.089429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42875 (local address 127.0.98.3:39133)
0924 19:31:38.089469 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.089470 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.089491 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:31:38.089518 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.089518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.089528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.089590 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:38.089790 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.090304 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.091229 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.091599 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.091668 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.091678 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.091835 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.091868 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:31:38.091985 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.089347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39133 (local address 127.0.98.1:42875)
0924 19:31:38.089392 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.089405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.089446 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.089601 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.089603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.089606 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:38.089695 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.089698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.090375 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.090376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.091140 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.091142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.091679 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.091680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.091770 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.091774 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.091823 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.091861 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":1}
I20260924 19:31:38.091990 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.089348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42363 (local address 127.0.98.1:33371)
0924 19:31:38.089531 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.089538 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.089584 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.089735 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.089736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.089739 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:38.089817 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.089818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.090415 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.090416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.091155 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.091157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.091657 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.091658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.091751 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.091756 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.091805 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.091843 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":56,"thread_start_us":65,"threads_started":1}
I20260924 19:31:38.092334 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc872c6fb646f6a8062b4da323b483" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb80b2c0f1e41608a9437427f8f08eb" is_pre_election: true
I20260924 19:31:38.092495 2552 raft_consensus.cc:2454] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 0.
I20260924 19:31:38.092501 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc872c6fb646f6a8062b4da323b483" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe786d7395b4d439c1484587c4f0448" is_pre_election: true
I20260924 19:31:38.092670 2412 raft_consensus.cc:2454] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 0.
I20260924 19:31:38.092732 2203 leader_election.cc:302] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d909237adf8645a190f96bbb1f9fcceb, edb80b2c0f1e41608a9437427f8f08eb; no voters:
I20260924 19:31:38.092908 2629 raft_consensus.cc:2781] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:38.092964 2629 raft_consensus.cc:482] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:38.092991 2629 raft_consensus.cc:3037] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.093658 2629 raft_consensus.cc:504] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.093766 2629 leader_election.cc:288] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 election: Requested vote from peers 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363), edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:38.093902 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc872c6fb646f6a8062b4da323b483" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe786d7395b4d439c1484587c4f0448"
I20260924 19:31:38.093928 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc872c6fb646f6a8062b4da323b483" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb80b2c0f1e41608a9437427f8f08eb"
I20260924 19:31:38.093966 2412 raft_consensus.cc:3037] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.094007 2552 raft_consensus.cc:3037] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.094650 2412 raft_consensus.cc:2454] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 1.
I20260924 19:31:38.094669 2552 raft_consensus.cc:2454] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 1.
I20260924 19:31:38.094799 2203 leader_election.cc:302] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d909237adf8645a190f96bbb1f9fcceb, edb80b2c0f1e41608a9437427f8f08eb; no voters:
I20260924 19:31:38.094890 2629 raft_consensus.cc:2781] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:38.095031 2629 raft_consensus.cc:686] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 1 LEADER]: Becoming Leader. State: Replica: d909237adf8645a190f96bbb1f9fcceb, State: Running, Role: LEADER
I20260924 19:31:38.095110 2629 consensus_queue.cc:237] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.095901 2114 catalog_manager.cc:5495] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb reported cstate change: term changed from 0 to 1, leader changed from <none> to d909237adf8645a190f96bbb1f9fcceb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } health_report { overall_health: UNKNOWN } } }
W20260924 19:31:38.099344 2319 tablet.cc:2369] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:31:38.154266 2599 tablet.cc:2369] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:31:38 dist-test-slave-m8jb krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:38.160199 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.153434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.0.98.62:43715)
0924 19:31:38.153497 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.153499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.153575 (+ 76us) server_negotiation.cc:407] Connection header received
0924 19:31:38.153629 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.153630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.153638 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.153684 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:38.154111 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.154737 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.155436 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.155625 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.155666 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.157864 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:38.157866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:38.157866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:38.157872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:38.159195 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:38.159601 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:38.159602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:38.159603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:38.159652 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:38.159812 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:38.159813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:38.159813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:38.159904 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:38.159944 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.160070 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.160138 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:31:38.160326 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.153344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43715 (local address 127.0.0.1:60820)
0924 19:31:38.153502 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.153514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.153582 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.153697 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.153697 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.153990 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:38.154058 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.154060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.154828 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.154829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.155370 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.155371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.155719 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.155721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.155773 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.156246 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:38.156251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:38.157765 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:38.159272 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:38.159279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:38.159281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:38.159524 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:38.159703 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:38.159704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:38.159704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:38.159765 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:38.159945 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:38.159950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:38.159995 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.160053 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.160201 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":54,"thread_start_us":75,"threads_started":1}
I20260924 19:31:38.162354 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60820:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:31:38.162537 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:38.163942 2392 tablet_service.cc:1467] Processing CreateTablet for tablet 3360236e3d444341aeb06e599828ce22 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c690fda0dc1647b0bc22efeedd1b2aec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:31:38.163949 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 3360236e3d444341aeb06e599828ce22 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c690fda0dc1647b0bc22efeedd1b2aec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:31:38.164093 2392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3360236e3d444341aeb06e599828ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:38.164101 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3360236e3d444341aeb06e599828ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:38.165308 2624 tablet_bootstrap.cc:492] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: Bootstrap starting.
I20260924 19:31:38.165313 2621 tablet_bootstrap.cc:492] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: Bootstrap starting.
I20260924 19:31:38.165408 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 3360236e3d444341aeb06e599828ce22 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c690fda0dc1647b0bc22efeedd1b2aec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:31:38.165557 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3360236e3d444341aeb06e599828ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:38.165722 2624 tablet_bootstrap.cc:654] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:38.166010 2621 tablet_bootstrap.cc:654] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:38.166585 2621 tablet_bootstrap.cc:492] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: No bootstrap required, opened a new log
I20260924 19:31:38.166627 2621 ts_tablet_manager.cc:1397] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:38.166759 2621 raft_consensus.cc:348] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.166833 2621 raft_consensus.cc:374] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:38.166852 2621 raft_consensus.cc:729] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe786d7395b4d439c1484587c4f0448, State: Initialized, Role: FOLLOWER
I20260924 19:31:38.166898 2621 consensus_queue.cc:260] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.166931 2622 tablet_bootstrap.cc:492] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: Bootstrap starting.
I20260924 19:31:38.166977 2621 ts_tablet_manager.cc:1428] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:38.167119 2627 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.167197 2627 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:38.167383 2622 tablet_bootstrap.cc:654] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:38.167990 2622 tablet_bootstrap.cc:492] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: No bootstrap required, opened a new log
I20260924 19:31:38.168033 2622 ts_tablet_manager.cc:1397] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:38.168165 2622 raft_consensus.cc:348] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.168224 2622 raft_consensus.cc:374] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:38.168246 2622 raft_consensus.cc:729] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb80b2c0f1e41608a9437427f8f08eb, State: Initialized, Role: FOLLOWER
I20260924 19:31:38.168294 2622 consensus_queue.cc:260] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.168378 2622 ts_tablet_manager.cc:1428] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:38.168455 2628 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.168547 2628 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:38.168660 2624 tablet_bootstrap.cc:492] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: No bootstrap required, opened a new log
I20260924 19:31:38.168713 2624 ts_tablet_manager.cc:1397] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:31:38.168854 2624 raft_consensus.cc:348] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.168911 2624 raft_consensus.cc:374] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:38.168927 2624 raft_consensus.cc:729] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d909237adf8645a190f96bbb1f9fcceb, State: Initialized, Role: FOLLOWER
I20260924 19:31:38.168967 2624 consensus_queue.cc:260] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.169047 2624 ts_tablet_manager.cc:1428] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:38.169158 2629 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.169251 2629 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:38.334414 2629 raft_consensus.cc:482] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:38.334524 2629 raft_consensus.cc:504] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.334682 2629 leader_election.cc:288] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363), edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:38.334890 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3360236e3d444341aeb06e599828ce22" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe786d7395b4d439c1484587c4f0448" is_pre_election: true
I20260924 19:31:38.334901 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3360236e3d444341aeb06e599828ce22" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb80b2c0f1e41608a9437427f8f08eb" is_pre_election: true
I20260924 19:31:38.334978 2412 raft_consensus.cc:2454] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 0.
I20260924 19:31:38.334998 2552 raft_consensus.cc:2454] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 0.
I20260924 19:31:38.335184 2205 leader_election.cc:302] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe786d7395b4d439c1484587c4f0448, d909237adf8645a190f96bbb1f9fcceb; no voters:
I20260924 19:31:38.335289 2629 raft_consensus.cc:2781] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:38.335330 2629 raft_consensus.cc:482] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:38.335345 2629 raft_consensus.cc:3037] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.335995 2629 raft_consensus.cc:504] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.336114 2629 leader_election.cc:288] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 election: Requested vote from peers 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363), edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:38.336251 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3360236e3d444341aeb06e599828ce22" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe786d7395b4d439c1484587c4f0448"
I20260924 19:31:38.336279 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3360236e3d444341aeb06e599828ce22" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb80b2c0f1e41608a9437427f8f08eb"
I20260924 19:31:38.336319 2412 raft_consensus.cc:3037] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.336344 2552 raft_consensus.cc:3037] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.336933 2412 raft_consensus.cc:2454] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 1.
I20260924 19:31:38.336949 2552 raft_consensus.cc:2454] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 1.
I20260924 19:31:38.337101 2203 leader_election.cc:302] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d909237adf8645a190f96bbb1f9fcceb, edb80b2c0f1e41608a9437427f8f08eb; no voters:
I20260924 19:31:38.337191 2629 raft_consensus.cc:2781] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:38.337252 2629 raft_consensus.cc:686] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [term 1 LEADER]: Becoming Leader. State: Replica: d909237adf8645a190f96bbb1f9fcceb, State: Running, Role: LEADER
I20260924 19:31:38.337303 2629 consensus_queue.cc:237] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.337632 2631 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: TxnStatusTablet state changed. Reason: New leader d909237adf8645a190f96bbb1f9fcceb. Latest consensus state: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.337711 2631 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:31:38.337903 2643 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:31:38.337919 2115 catalog_manager.cc:5495] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb reported cstate change: term changed from 0 to 1, leader changed from <none> to d909237adf8645a190f96bbb1f9fcceb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } health_report { overall_health: UNKNOWN } } }
I20260924 19:31:38.338013 2643 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:31:38.351171 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.347983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.0.1:59864)
0924 19:31:38.348061 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.348077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.348112 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.348268 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.348269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.348270 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:38.348316 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.348318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.349241 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.349242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.349938 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.349942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.350515 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.350516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.350595 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.350619 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.350928 (+ 309us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.350936 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.350992 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.351066 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 19:31:38.351184 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.348042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.98.1:45389)
0924 19:31:38.348088 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.348090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.348130 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:31:38.348177 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.348178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.348199 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.348266 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:38.348365 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.349142 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.350041 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.350375 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.350445 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.350732 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.350862 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.350937 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.351074 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.351100 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:31:38.353552 2552 raft_consensus.cc:1261] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Refusing update from remote peer d909237adf8645a190f96bbb1f9fcceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:38.353555 2412 raft_consensus.cc:1261] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Refusing update from remote peer d909237adf8645a190f96bbb1f9fcceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:38.353956 2631 consensus_queue.cc:1035] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:38.354209 2629 consensus_queue.cc:1035] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:38.355263 2627 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: TxnStatusTablet state changed. Reason: New leader d909237adf8645a190f96bbb1f9fcceb. Latest consensus state: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.355362 2627 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:38.355289 2628 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: TxnStatusTablet state changed. Reason: New leader d909237adf8645a190f96bbb1f9fcceb. Latest consensus state: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.355404 2628 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:38.356436 2629 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.356525 2629 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:31:38.356598 2646 mvcc.cc:205] Tried to move back new op lower bound from 7332979910052868096 to 7332979909989900288. Current Snapshot: MvccSnapshot[applied={T|T < 7332979910052868096}]
I20260924 19:31:38.356701 2631 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.356791 2631 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P d909237adf8645a190f96bbb1f9fcceb: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:31:38.357275 2627 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.357360 2627 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P 4fe786d7395b4d439c1484587c4f0448: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:38.357404 2647 mvcc.cc:205] Tried to move back new op lower bound from 7332979910052868096 to 7332979909989900288. Current Snapshot: MvccSnapshot[applied={T|T < 7332979910052868096}]
I20260924 19:31:38.357372 2628 tablet_replica.cc:439] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } } }
I20260924 19:31:38.357472 2628 tablet_replica.cc:442] T 3360236e3d444341aeb06e599828ce22 P edb80b2c0f1e41608a9437427f8f08eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:31:38.357920 2648 mvcc.cc:205] Tried to move back new op lower bound from 7332979910052868096 to 7332979909989900288. Current Snapshot: MvccSnapshot[applied={T|T < 7332979910052868096 or (T in {7332979910052868096})}]
I20260924 19:31:38.361970 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.359721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.0.98.1:45389)
0924 19:31:38.359757 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.359759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.359768 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:38.359825 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.359826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.359835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.359898 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:38.360045 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.360510 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.361334 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.361543 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.361604 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.361741 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.361774 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.361816 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.361881 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.361899 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:31:38.367118 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.364683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42133 (local address 127.0.98.1:45389)
0924 19:31:38.364719 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.364721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.364850 (+ 129us) server_negotiation.cc:407] Connection header received
0924 19:31:38.364902 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.364903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.364911 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.364966 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:38.365065 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.365598 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.366468 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.366658 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.366711 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.366893 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.366923 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.366964 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.367011 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.367045 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":18}
I20260924 19:31:38.367174 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.364605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.98.1:42133)
0924 19:31:38.364778 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.364786 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.364850 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.364953 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.364954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.364957 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:38.365020 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.365022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.365676 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.365676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.366406 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.366408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.366732 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.366733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.366808 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.366820 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.366969 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.366970 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.367010 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.367061 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260924 19:31:38.369462 2412 raft_consensus.cc:1261] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Refusing update from remote peer d909237adf8645a190f96bbb1f9fcceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:38.369462 2552 raft_consensus.cc:1261] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Refusing update from remote peer d909237adf8645a190f96bbb1f9fcceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:38.369681 2658 consensus_queue.cc:1035] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:38.369792 2629 consensus_queue.cc:1035] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:38.380891 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:38.381009 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:38.381734 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: Sending DeleteTablet for 3 replicas of tablet c4dc872c6fb646f6a8062b4da323b483
I20260924 19:31:38.381982 2392 tablet_service.cc:1514] Processing DeleteTablet for tablet c4dc872c6fb646f6a8062b4da323b483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260924 19:31:38.382012 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet c4dc872c6fb646f6a8062b4da323b483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39026
I20260924 19:31:38.382171 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet c4dc872c6fb646f6a8062b4da323b483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260924 19:31:38.382328 2663 tablet_replica.cc:330] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: stopping tablet replica
I20260924 19:31:38.382305 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet c4dc872c6fb646f6a8062b4da323b483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59850
I20260924 19:31:38.382426 2663 raft_consensus.cc:2229] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:38.382515 2663 raft_consensus.cc:2258] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:38.382606 2664 tablet_replica.cc:330] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: stopping tablet replica
I20260924 19:31:38.382642 2662 tablet_replica.cc:330] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: stopping tablet replica
I20260924 19:31:38.382606 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:38.382754 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:38.382752 2662 raft_consensus.cc:2229] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:38.382864 2662 raft_consensus.cc:2258] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:38.382979 2664 raft_consensus.cc:2229] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:31:38.383139 2664 raft_consensus.cc:2258] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:38.383385 2662 ts_tablet_manager.cc:1905] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:38.384057 2391 tablet_service.cc:1467] Processing CreateTablet for tablet 17673df6472049dea427a221689c4c36 (DEFAULT_TABLE table=test-table [id=70e0a50251914bc0a5f27ef1b15d951e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:38.384114 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 17673df6472049dea427a221689c4c36 (DEFAULT_TABLE table=test-table [id=70e0a50251914bc0a5f27ef1b15d951e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:38.384171 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17673df6472049dea427a221689c4c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:38.384208 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17673df6472049dea427a221689c4c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:38.385521 2624 tablet_bootstrap.cc:492] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: Bootstrap starting.
I20260924 19:31:38.385527 2621 tablet_bootstrap.cc:492] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: Bootstrap starting.
I20260924 19:31:38.385895 2621 tablet_bootstrap.cc:654] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:38.385903 2624 tablet_bootstrap.cc:654] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:38.386353 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 17673df6472049dea427a221689c4c36 (DEFAULT_TABLE table=test-table [id=70e0a50251914bc0a5f27ef1b15d951e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:38.386442 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17673df6472049dea427a221689c4c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:38.383634 2664 ts_tablet_manager.cc:1905] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:38.387480 2662 ts_tablet_manager.cc:1918] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:38.387555 2662 log.cc:1196] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/c4dc872c6fb646f6a8062b4da323b483
I20260924 19:31:38.387686 2622 tablet_bootstrap.cc:492] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: Bootstrap starting.
I20260924 19:31:38.387801 2662 ts_tablet_manager.cc:1939] T c4dc872c6fb646f6a8062b4da323b483 P edb80b2c0f1e41608a9437427f8f08eb: Deleting consensus metadata
I20260924 19:31:38.388082 2622 tablet_bootstrap.cc:654] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:38.388365 2621 tablet_bootstrap.cc:492] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: No bootstrap required, opened a new log
I20260924 19:31:38.388414 2621 ts_tablet_manager.cc:1397] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:31:38.388379 2664 ts_tablet_manager.cc:1918] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:38.388553 2099 catalog_manager.cc:4841] TS edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133): tablet c4dc872c6fb646f6a8062b4da323b483 (table test-table [id=455f8f0d722045cda18e06a82616442d]) successfully deleted
I20260924 19:31:38.388550 2621 raft_consensus.cc:348] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.388612 2621 raft_consensus.cc:374] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:38.388679 2621 raft_consensus.cc:729] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe786d7395b4d439c1484587c4f0448, State: Initialized, Role: FOLLOWER
I20260924 19:31:38.388577 2664 log.cc:1196] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/c4dc872c6fb646f6a8062b4da323b483
I20260924 19:31:38.388726 2621 consensus_queue.cc:260] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.388813 2621 ts_tablet_manager.cc:1428] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:38.389099 2664 ts_tablet_manager.cc:1939] T c4dc872c6fb646f6a8062b4da323b483 P d909237adf8645a190f96bbb1f9fcceb: Deleting consensus metadata
I20260924 19:31:38.389150 2663 ts_tablet_manager.cc:1905] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:38.389524 2101 catalog_manager.cc:4841] TS d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389): tablet c4dc872c6fb646f6a8062b4da323b483 (table test-table [id=455f8f0d722045cda18e06a82616442d]) successfully deleted
I20260924 19:31:38.389906 2622 tablet_bootstrap.cc:492] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: No bootstrap required, opened a new log
I20260924 19:31:38.389916 2624 tablet_bootstrap.cc:492] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: No bootstrap required, opened a new log
I20260924 19:31:38.389950 2622 ts_tablet_manager.cc:1397] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:38.389955 2624 ts_tablet_manager.cc:1397] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 19:31:38.390071 2622 raft_consensus.cc:348] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.390087 2624 raft_consensus.cc:348] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.390129 2622 raft_consensus.cc:374] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:38.390149 2622 raft_consensus.cc:729] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb80b2c0f1e41608a9437427f8f08eb, State: Initialized, Role: FOLLOWER
I20260924 19:31:38.390148 2624 raft_consensus.cc:374] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:38.390170 2624 raft_consensus.cc:729] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d909237adf8645a190f96bbb1f9fcceb, State: Initialized, Role: FOLLOWER
I20260924 19:31:38.390190 2622 consensus_queue.cc:260] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.390208 2624 consensus_queue.cc:260] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.390265 2622 ts_tablet_manager.cc:1428] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:38.390290 2624 ts_tablet_manager.cc:1428] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:38.390798 2663 ts_tablet_manager.cc:1918] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:38.390864 2663 log.cc:1196] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/c4dc872c6fb646f6a8062b4da323b483
I20260924 19:31:38.391103 2663 ts_tablet_manager.cc:1939] T c4dc872c6fb646f6a8062b4da323b483 P 4fe786d7395b4d439c1484587c4f0448: Deleting consensus metadata
I20260924 19:31:38.391613 2101 catalog_manager.cc:4841] TS 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363): tablet c4dc872c6fb646f6a8062b4da323b483 (table test-table [id=455f8f0d722045cda18e06a82616442d]) successfully deleted
W20260924 19:31:38.391742 2101 catalog_manager.cc:4808] TS 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363): delete failed for tablet c4dc872c6fb646f6a8062b4da323b483 because the tablet was not found. No further retry: Not found: Tablet not found: c4dc872c6fb646f6a8062b4da323b483
I20260924 19:31:38.402130 2627 raft_consensus.cc:482] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:38.402211 2627 raft_consensus.cc:504] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.402518 2627 leader_election.cc:288] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389), edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:38.405890 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.402888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40019 (local address 127.0.98.3:39133)
0924 19:31:38.402928 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.402930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.402939 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:38.403012 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.403013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.403021 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.403074 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:38.403211 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.403894 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.405132 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.405531 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.405588 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.405599 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.405779 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.405823 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:31:38.405990 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.402712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33993 (local address 127.0.98.1:45389)
0924 19:31:38.402764 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.402766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.402945 (+ 179us) server_negotiation.cc:407] Connection header received
0924 19:31:38.403051 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.403052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.403063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.403104 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:38.403241 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.403703 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.405108 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.405670 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.405751 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.405761 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.405892 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.405924 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:31:38.406008 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.402673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.98.2:33993)
0924 19:31:38.402883 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.402894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.402964 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.403140 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.403141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.403142 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:38.403194 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.403195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.403777 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.403778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.405022 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.405025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.405770 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.405771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.405829 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.405832 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.405881 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.405918 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":38,"thread_start_us":57,"threads_started":1}
I20260924 19:31:38.406265 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.402697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39133 (local address 127.0.98.2:40019)
0924 19:31:38.402884 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.402894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.402947 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.403077 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.403078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.403079 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:38.403150 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.403152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.403970 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.403971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.405075 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.405077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.405631 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.405632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.405708 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.405713 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.405959 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.405990 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
I20260924 19:31:38.406252 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17673df6472049dea427a221689c4c36" candidate_uuid: "4fe786d7395b4d439c1484587c4f0448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d909237adf8645a190f96bbb1f9fcceb" is_pre_election: true
I20260924 19:31:38.406360 2272 raft_consensus.cc:2454] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe786d7395b4d439c1484587c4f0448 in term 0.
I20260924 19:31:38.406509 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17673df6472049dea427a221689c4c36" candidate_uuid: "4fe786d7395b4d439c1484587c4f0448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb80b2c0f1e41608a9437427f8f08eb" is_pre_election: true
I20260924 19:31:38.406597 2552 raft_consensus.cc:2454] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fe786d7395b4d439c1484587c4f0448 in term 0.
I20260924 19:31:38.406569 2345 leader_election.cc:302] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe786d7395b4d439c1484587c4f0448, d909237adf8645a190f96bbb1f9fcceb; no voters:
I20260924 19:31:38.406714 2627 raft_consensus.cc:2781] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:38.406754 2627 raft_consensus.cc:482] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:38.406781 2627 raft_consensus.cc:3037] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.407431 2627 raft_consensus.cc:504] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.407542 2627 leader_election.cc:288] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [CANDIDATE]: Term 1 election: Requested vote from peers d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389), edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:38.407727 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17673df6472049dea427a221689c4c36" candidate_uuid: "4fe786d7395b4d439c1484587c4f0448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d909237adf8645a190f96bbb1f9fcceb"
I20260924 19:31:38.407738 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17673df6472049dea427a221689c4c36" candidate_uuid: "4fe786d7395b4d439c1484587c4f0448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb80b2c0f1e41608a9437427f8f08eb"
I20260924 19:31:38.407799 2272 raft_consensus.cc:3037] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.407799 2552 raft_consensus.cc:3037] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:38.408463 2272 raft_consensus.cc:2454] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe786d7395b4d439c1484587c4f0448 in term 1.
I20260924 19:31:38.408463 2552 raft_consensus.cc:2454] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fe786d7395b4d439c1484587c4f0448 in term 1.
I20260924 19:31:38.408614 2345 leader_election.cc:302] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe786d7395b4d439c1484587c4f0448, d909237adf8645a190f96bbb1f9fcceb; no voters:
I20260924 19:31:38.408715 2627 raft_consensus.cc:2781] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:38.408831 2627 raft_consensus.cc:686] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 1 LEADER]: Becoming Leader. State: Replica: 4fe786d7395b4d439c1484587c4f0448, State: Running, Role: LEADER
I20260924 19:31:38.408901 2627 consensus_queue.cc:237] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:38.409564 2114 catalog_manager.cc:5495] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4fe786d7395b4d439c1484587c4f0448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } health_report { overall_health: UNKNOWN } } }
I20260924 19:31:38.416139 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.413673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.0.98.2:42363)
0924 19:31:38.413716 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.413717 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.413723 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:31:38.413746 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.413747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.413760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.413814 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:38.413941 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.414486 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.415458 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.415684 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.415730 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.415882 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.415971 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.416022 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.416070 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.416087 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:31:38.420823 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.418305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47173 (local address 127.0.98.1:45389)
0924 19:31:38.418346 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.418347 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.418472 (+ 125us) server_negotiation.cc:407] Connection header received
0924 19:31:38.418530 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.418530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.418537 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.418576 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:38.418710 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.419142 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.420170 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.420340 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.420404 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.420581 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.420606 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.420681 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.420735 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.420751 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:31:38.420863 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.418218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.98.2:47173)
0924 19:31:38.418402 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.418413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.418477 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.418583 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.418583 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.418585 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:38.418645 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.418646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.419217 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.419218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.420096 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.420098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.420429 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.420430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.420492 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.420507 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.420687 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.420688 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.420734 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.420777 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260924 19:31:38.423115 2552 raft_consensus.cc:1261] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Refusing update from remote peer 4fe786d7395b4d439c1484587c4f0448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:38.423207 2272 raft_consensus.cc:1261] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Refusing update from remote peer 4fe786d7395b4d439c1484587c4f0448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:38.423592 2670 consensus_queue.cc:1035] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:38.423755 2627 consensus_queue.cc:1035] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:38.431419 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.429219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54553 (local address 127.0.98.2:42363)
0924 19:31:38.429247 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:38.429249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:38.429256 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:31:38.429273 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:38.429274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:38.429284 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:38.429327 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:38.429465 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.429930 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.430665 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.430917 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.430976 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.431192 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.431226 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.431269 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:38.431311 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 19:31:38.431350 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 19:31:38.431420 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:38.428965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42363 (local address 127.0.98.1:54553)
0924 19:31:38.429002 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:38.429013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:38.429048 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:38.429367 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:38.429369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:38.429370 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:38.429407 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.429408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.430001 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.430002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.430595 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:38.430597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:38.431067 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:38.431068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:38.431127 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:38.431133 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:38.431270 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:38.431272 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:31:38.431310 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 19:31:38.431353 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 19:31:38.431866 2647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:31:39.948670 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:39.948768 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:39.949533 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: Sending DeleteTablet for 3 replicas of tablet 17673df6472049dea427a221689c4c36
I20260924 19:31:39.949759 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 17673df6472049dea427a221689c4c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59850
I20260924 19:31:39.949774 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 17673df6472049dea427a221689c4c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260924 19:31:39.949941 2701 tablet_replica.cc:330] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: stopping tablet replica
I20260924 19:31:39.949939 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 17673df6472049dea427a221689c4c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39026
I20260924 19:31:39.950002 2701 raft_consensus.cc:2229] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:39.950055 2701 raft_consensus.cc:2258] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:39.950153 2700 tablet_replica.cc:330] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: stopping tablet replica
I20260924 19:31:39.950206 2700 raft_consensus.cc:2229] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:31:39.950289 2700 raft_consensus.cc:2258] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:39.950310 2701 ts_tablet_manager.cc:1905] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:39.950484 2700 ts_tablet_manager.cc:1905] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:39.952001 2701 ts_tablet_manager.cc:1918] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:31:39.952010 2700 ts_tablet_manager.cc:1918] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:31:39.952059 2700 log.cc:1196] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/17673df6472049dea427a221689c4c36
I20260924 19:31:39.952122 2702 tablet_replica.cc:330] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: stopping tablet replica
I20260924 19:31:39.952191 2702 raft_consensus.cc:2229] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:39.952239 2702 raft_consensus.cc:2258] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:39.952319 2700 ts_tablet_manager.cc:1939] T 17673df6472049dea427a221689c4c36 P 4fe786d7395b4d439c1484587c4f0448: Deleting consensus metadata
I20260924 19:31:39.952493 2702 ts_tablet_manager.cc:1905] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:39.952605 2101 catalog_manager.cc:4841] TS 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363): tablet 17673df6472049dea427a221689c4c36 (table test-table [id=70e0a50251914bc0a5f27ef1b15d951e]) successfully deleted
I20260924 19:31:39.953790 2702 ts_tablet_manager.cc:1918] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:31:39.953836 2702 log.cc:1196] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/17673df6472049dea427a221689c4c36
I20260924 19:31:39.954001 2702 ts_tablet_manager.cc:1939] T 17673df6472049dea427a221689c4c36 P edb80b2c0f1e41608a9437427f8f08eb: Deleting consensus metadata
I20260924 19:31:39.954272 2099 catalog_manager.cc:4841] TS edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133): tablet 17673df6472049dea427a221689c4c36 (table test-table [id=70e0a50251914bc0a5f27ef1b15d951e]) successfully deleted
I20260924 19:31:39.954890 2701 log.cc:1196] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/17673df6472049dea427a221689c4c36
I20260924 19:31:39.955080 2701 ts_tablet_manager.cc:1939] T 17673df6472049dea427a221689c4c36 P d909237adf8645a190f96bbb1f9fcceb: Deleting consensus metadata
I20260924 19:31:39.955351 2101 catalog_manager.cc:4841] TS d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389): tablet 17673df6472049dea427a221689c4c36 (table test-table [id=70e0a50251914bc0a5f27ef1b15d951e]) successfully deleted
Sep 24 19:31:39 dist-test-slave-m8jb krb5kdc[2070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278299, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:31:39 dist-test-slave-m8jb krb5kdc[2070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278299, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:39.973506 2710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:39.966738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.98.62:43715)
0924 19:31:39.966865 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:39.966867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:39.966922 (+ 55us) server_negotiation.cc:407] Connection header received
0924 19:31:39.966977 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:39.966978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:39.966987 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:39.967044 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:39.967564 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:39.968064 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:39.968863 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:39.969056 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:39.969101 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:39.971224 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:39.971225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:39.971225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:39.971232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:39.972420 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:39.972790 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:39.972791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:39.972792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:39.972847 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:39.973008 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:39.973009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:39.973009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:39.973120 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:39.973219 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:39.973319 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 19:31:39.973405 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260924 19:31:39.974788 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:39.974931 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:39.976145 2532 tablet_service.cc:1467] Processing CreateTablet for tablet 1cef495cbdd24d7c8ade4080ba70a6c2 (DEFAULT_TABLE table=test-table [id=4211a588f2cf4e519bffab75920a6431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:39.976153 2391 tablet_service.cc:1467] Processing CreateTablet for tablet 1cef495cbdd24d7c8ade4080ba70a6c2 (DEFAULT_TABLE table=test-table [id=4211a588f2cf4e519bffab75920a6431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:39.976153 2252 tablet_service.cc:1467] Processing CreateTablet for tablet 1cef495cbdd24d7c8ade4080ba70a6c2 (DEFAULT_TABLE table=test-table [id=4211a588f2cf4e519bffab75920a6431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:39.976277 2252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cef495cbdd24d7c8ade4080ba70a6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:39.976276 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cef495cbdd24d7c8ade4080ba70a6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:39.976287 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cef495cbdd24d7c8ade4080ba70a6c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:39.977656 2713 tablet_bootstrap.cc:492] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: Bootstrap starting.
I20260924 19:31:39.977777 2711 tablet_bootstrap.cc:492] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: Bootstrap starting.
I20260924 19:31:39.977990 2713 tablet_bootstrap.cc:654] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:39.978103 2711 tablet_bootstrap.cc:654] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:39.978286 2712 tablet_bootstrap.cc:492] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: Bootstrap starting.
I20260924 19:31:39.978528 2713 tablet_bootstrap.cc:492] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: No bootstrap required, opened a new log
I20260924 19:31:39.978580 2713 ts_tablet_manager.cc:1397] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:39.978588 2712 tablet_bootstrap.cc:654] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:39.978662 2711 tablet_bootstrap.cc:492] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: No bootstrap required, opened a new log
I20260924 19:31:39.978688 2713 raft_consensus.cc:348] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:39.978729 2711 ts_tablet_manager.cc:1397] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:39.978760 2713 raft_consensus.cc:374] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:39.978785 2713 raft_consensus.cc:729] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d909237adf8645a190f96bbb1f9fcceb, State: Initialized, Role: FOLLOWER
I20260924 19:31:39.978822 2713 consensus_queue.cc:260] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:39.978873 2711 raft_consensus.cc:348] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:39.978938 2711 raft_consensus.cc:374] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:39.978955 2711 raft_consensus.cc:729] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb80b2c0f1e41608a9437427f8f08eb, State: Initialized, Role: FOLLOWER
I20260924 19:31:39.978994 2711 consensus_queue.cc:260] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:39.979039 2713 ts_tablet_manager.cc:1428] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:39.979122 2711 ts_tablet_manager.cc:1428] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:39.979146 2712 tablet_bootstrap.cc:492] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: No bootstrap required, opened a new log
I20260924 19:31:39.979183 2712 ts_tablet_manager.cc:1397] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:39.979275 2712 raft_consensus.cc:348] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:39.979373 2712 raft_consensus.cc:374] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:39.979405 2712 raft_consensus.cc:729] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe786d7395b4d439c1484587c4f0448, State: Initialized, Role: FOLLOWER
I20260924 19:31:39.979462 2712 consensus_queue.cc:260] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:39.979592 2712 ts_tablet_manager.cc:1428] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:40.063421 2717 raft_consensus.cc:482] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:40.063510 2717 raft_consensus.cc:504] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.063817 2717 leader_election.cc:288] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389), 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363)
I20260924 19:31:40.068075 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.064236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52155 (local address 127.0.98.2:42363)
0924 19:31:40.064376 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:40.064377 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:40.064385 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:40.064407 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:40.064408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:40.064418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:40.064486 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:40.064688 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.065478 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.067074 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.067680 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.067772 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.067787 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:40.067966 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 19:31:40.068008 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260924 19:31:40.068073 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.064033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37213 (local address 127.0.98.1:45389)
0924 19:31:40.064369 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:40.064371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:40.064379 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:40.064407 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:40.064408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:40.064418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:40.064486 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:40.064679 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.065483 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.067074 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.067725 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.067792 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.067804 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:40.067948 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:31:40.067989 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":100,"threads_started":1}
I20260924 19:31:40.068094 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.063973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.98.3:37213)
0924 19:31:40.064130 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:40.064142 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:40.064206 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:40.064502 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:40.064502 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:40.064504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:40.064593 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:40.064596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.065545 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.065546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:40.066966 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:40.066968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.067800 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.067801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:40.067880 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.067882 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:31:40.067934 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:31:40.067988 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":49,"thread_start_us":85,"threads_started":1}
I20260924 19:31:40.068102 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.064149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42363 (local address 127.0.98.3:52155)
0924 19:31:40.064262 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:40.064271 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:40.064321 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:40.064500 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:40.064500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:40.064503 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:40.064593 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:40.064595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.065547 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.065548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:40.066966 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:40.066968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.067798 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.067799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:40.067871 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.067876 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:31:40.067934 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:31:40.067988 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":87,"thread_start_us":54,"threads_started":1}
I20260924 19:31:40.068327 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cef495cbdd24d7c8ade4080ba70a6c2" candidate_uuid: "edb80b2c0f1e41608a9437427f8f08eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d909237adf8645a190f96bbb1f9fcceb" is_pre_election: true
I20260924 19:31:40.068415 2272 raft_consensus.cc:2454] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edb80b2c0f1e41608a9437427f8f08eb in term 0.
I20260924 19:31:40.068584 2485 leader_election.cc:302] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d909237adf8645a190f96bbb1f9fcceb, edb80b2c0f1e41608a9437427f8f08eb; no voters:
I20260924 19:31:40.068732 2717 raft_consensus.cc:2781] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:40.068787 2717 raft_consensus.cc:482] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:40.068816 2717 raft_consensus.cc:3037] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:40.069124 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cef495cbdd24d7c8ade4080ba70a6c2" candidate_uuid: "edb80b2c0f1e41608a9437427f8f08eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe786d7395b4d439c1484587c4f0448" is_pre_election: true
I20260924 19:31:40.069217 2412 raft_consensus.cc:2454] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edb80b2c0f1e41608a9437427f8f08eb in term 0.
I20260924 19:31:40.069609 2717 raft_consensus.cc:504] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.069730 2717 leader_election.cc:288] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [CANDIDATE]: Term 1 election: Requested vote from peers d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389), 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363)
I20260924 19:31:40.069900 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cef495cbdd24d7c8ade4080ba70a6c2" candidate_uuid: "edb80b2c0f1e41608a9437427f8f08eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe786d7395b4d439c1484587c4f0448"
I20260924 19:31:40.069898 2272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cef495cbdd24d7c8ade4080ba70a6c2" candidate_uuid: "edb80b2c0f1e41608a9437427f8f08eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d909237adf8645a190f96bbb1f9fcceb"
I20260924 19:31:40.069988 2272 raft_consensus.cc:3037] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:40.069986 2412 raft_consensus.cc:3037] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:40.070710 2272 raft_consensus.cc:2454] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edb80b2c0f1e41608a9437427f8f08eb in term 1.
I20260924 19:31:40.070710 2412 raft_consensus.cc:2454] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edb80b2c0f1e41608a9437427f8f08eb in term 1.
I20260924 19:31:40.070859 2485 leader_election.cc:302] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fe786d7395b4d439c1484587c4f0448, edb80b2c0f1e41608a9437427f8f08eb; no voters:
I20260924 19:31:40.070940 2717 raft_consensus.cc:2781] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:40.071089 2717 raft_consensus.cc:686] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 1 LEADER]: Becoming Leader. State: Replica: edb80b2c0f1e41608a9437427f8f08eb, State: Running, Role: LEADER
I20260924 19:31:40.071158 2717 consensus_queue.cc:237] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.071846 2114 catalog_manager.cc:5495] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb reported cstate change: term changed from 0 to 1, leader changed from <none> to edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "edb80b2c0f1e41608a9437427f8f08eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } health_report { overall_health: HEALTHY } } }
I20260924 19:31:40.096619 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.094178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.0.98.3:39133)
0924 19:31:40.094325 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:40.094327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:40.094333 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:31:40.094369 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:40.094369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:40.094390 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:40.094459 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:40.094583 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.095084 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.095837 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.096058 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.096111 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.096265 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:40.096378 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:40.096443 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:40.096486 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:31:40.096506 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260924 19:31:40.101135 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.098969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45611 (local address 127.0.98.1:45389)
0924 19:31:40.099025 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:40.099027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:40.099049 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:31:40.099071 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:40.099072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:40.099084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:40.099124 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:40.099266 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.099738 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.100479 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.100700 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.100754 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.100902 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:40.100934 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:40.100974 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:40.101039 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:31:40.101060 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:31:40.101162 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.098823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.98.3:45611)
0924 19:31:40.098961 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:40.098972 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:40.099021 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:40.099125 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:40.099125 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:40.099127 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:40.099207 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:40.099208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.099810 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.099811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:40.100404 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:40.100405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.100758 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.100759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:40.100822 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.100836 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:40.100994 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:40.100996 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:31:40.101029 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 19:31:40.101073 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260924 19:31:40.103060 2272 raft_consensus.cc:1261] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Refusing update from remote peer edb80b2c0f1e41608a9437427f8f08eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:40.103067 2412 raft_consensus.cc:1261] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Refusing update from remote peer edb80b2c0f1e41608a9437427f8f08eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:40.103303 2722 consensus_queue.cc:1035] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:40.103428 2717 consensus_queue.cc:1035] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:40.109731 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.107585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57731 (local address 127.0.98.3:39133)
0924 19:31:40.107621 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:40.107623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:40.107722 (+ 99us) server_negotiation.cc:407] Connection header received
0924 19:31:40.107777 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:40.107778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:40.107789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:40.107843 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:40.107920 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.108407 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.109088 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.109331 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.109388 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.109518 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:40.109550 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:40.109594 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:40.109646 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 19:31:40.109669 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:40.109752 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.107529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39133 (local address 127.0.98.1:57731)
0924 19:31:40.107668 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:40.107678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:40.107730 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:40.107845 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:40.107846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:40.107847 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:31:40.107886 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:40.107887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.108457 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.108458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:40.109030 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:40.109031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.109402 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.109403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:40.109461 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.109466 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:31:40.109598 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:31:40.109601 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:40.109642 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:31:40.109689 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260924 19:31:40.115727 2114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:40.115820 2114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:40.116393 2114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: Sending DeleteTablet for 3 replicas of tablet 1cef495cbdd24d7c8ade4080ba70a6c2
I20260924 19:31:40.116655 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cef495cbdd24d7c8ade4080ba70a6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260924 19:31:40.116707 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cef495cbdd24d7c8ade4080ba70a6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39026
I20260924 19:31:40.116748 2700 tablet_replica.cc:330] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: stopping tablet replica
I20260924 19:31:40.116791 2702 tablet_replica.cc:330] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: stopping tablet replica
I20260924 19:31:40.116794 2700 raft_consensus.cc:2229] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:40.116828 2700 raft_consensus.cc:2258] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:40.116833 2702 raft_consensus.cc:2229] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:31:40.116902 2702 raft_consensus.cc:2258] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:40.116850 2114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:40.116979 2114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:40.117183 2702 ts_tablet_manager.cc:1905] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:40.117250 2700 ts_tablet_manager.cc:1905] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:40.118671 2702 ts_tablet_manager.cc:1918] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:40.118729 2702 log.cc:1196] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/1cef495cbdd24d7c8ade4080ba70a6c2
I20260924 19:31:40.118919 2702 ts_tablet_manager.cc:1939] T 1cef495cbdd24d7c8ade4080ba70a6c2 P edb80b2c0f1e41608a9437427f8f08eb: Deleting consensus metadata
I20260924 19:31:40.119103 2251 tablet_service.cc:1467] Processing CreateTablet for tablet b6cf512f61b74834981b8e8d44ae4a07 (DEFAULT_TABLE table=test-table [id=94acc336463049ae956f87be866b0d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:40.119170 2532 tablet_service.cc:1467] Processing CreateTablet for tablet b6cf512f61b74834981b8e8d44ae4a07 (DEFAULT_TABLE table=test-table [id=94acc336463049ae956f87be866b0d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:40.119207 2251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cf512f61b74834981b8e8d44ae4a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:40.119253 2532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cf512f61b74834981b8e8d44ae4a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:40.120126 2713 tablet_bootstrap.cc:492] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb: Bootstrap starting.
I20260924 19:31:40.120285 2391 tablet_service.cc:1467] Processing CreateTablet for tablet b6cf512f61b74834981b8e8d44ae4a07 (DEFAULT_TABLE table=test-table [id=94acc336463049ae956f87be866b0d63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:40.120340 2711 tablet_bootstrap.cc:492] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: Bootstrap starting.
I20260924 19:31:40.120441 2713 tablet_bootstrap.cc:654] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:40.120694 2711 tablet_bootstrap.cc:654] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:40.121071 2252 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cef495cbdd24d7c8ade4080ba70a6c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59850
I20260924 19:31:40.121264 2711 tablet_bootstrap.cc:492] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: No bootstrap required, opened a new log
I20260924 19:31:40.121312 2711 ts_tablet_manager.cc:1397] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:40.121343 2700 ts_tablet_manager.cc:1918] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:40.121352 2391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6cf512f61b74834981b8e8d44ae4a07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:40.121398 2700 log.cc:1196] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/1cef495cbdd24d7c8ade4080ba70a6c2
I20260924 19:31:40.121469 2711 raft_consensus.cc:348] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.121537 2711 raft_consensus.cc:374] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:40.121556 2711 raft_consensus.cc:729] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb80b2c0f1e41608a9437427f8f08eb, State: Initialized, Role: FOLLOWER
I20260924 19:31:40.121591 2700 ts_tablet_manager.cc:1939] T 1cef495cbdd24d7c8ade4080ba70a6c2 P 4fe786d7395b4d439c1484587c4f0448: Deleting consensus metadata
I20260924 19:31:40.121595 2711 consensus_queue.cc:260] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.121713 2711 ts_tablet_manager.cc:1428] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:40.121789 2701 tablet_replica.cc:330] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: stopping tablet replica
I20260924 19:31:40.121840 2701 raft_consensus.cc:2229] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:40.121878 2701 raft_consensus.cc:2258] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:40.121865 2101 catalog_manager.cc:4841] TS 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363): tablet 1cef495cbdd24d7c8ade4080ba70a6c2 (table test-table [id=4211a588f2cf4e519bffab75920a6431]) successfully deleted
I20260924 19:31:40.122313 2701 ts_tablet_manager.cc:1905] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:40.122380 2099 catalog_manager.cc:4841] TS edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133): tablet 1cef495cbdd24d7c8ade4080ba70a6c2 (table test-table [id=4211a588f2cf4e519bffab75920a6431]) successfully deleted
I20260924 19:31:40.122586 2712 tablet_bootstrap.cc:492] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448: Bootstrap starting.
I20260924 19:31:40.122788 2713 tablet_bootstrap.cc:492] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb: No bootstrap required, opened a new log
I20260924 19:31:40.122820 2713 ts_tablet_manager.cc:1397] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260924 19:31:40.122857 2712 tablet_bootstrap.cc:654] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:40.122892 2713 raft_consensus.cc:348] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.122934 2713 raft_consensus.cc:374] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:40.122946 2713 raft_consensus.cc:729] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d909237adf8645a190f96bbb1f9fcceb, State: Initialized, Role: FOLLOWER
I20260924 19:31:40.122979 2713 consensus_queue.cc:260] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.123056 2713 ts_tablet_manager.cc:1428] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:40.123451 2712 tablet_bootstrap.cc:492] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448: No bootstrap required, opened a new log
I20260924 19:31:40.123492 2712 ts_tablet_manager.cc:1397] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:40.123584 2712 raft_consensus.cc:348] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.123646 2712 raft_consensus.cc:374] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:40.123663 2712 raft_consensus.cc:729] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fe786d7395b4d439c1484587c4f0448, State: Initialized, Role: FOLLOWER
I20260924 19:31:40.123704 2712 consensus_queue.cc:260] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.123785 2712 ts_tablet_manager.cc:1428] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:40.123843 2701 ts_tablet_manager.cc:1918] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:31:40.123886 2701 log.cc:1196] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/1cef495cbdd24d7c8ade4080ba70a6c2
I20260924 19:31:40.124069 2701 ts_tablet_manager.cc:1939] T 1cef495cbdd24d7c8ade4080ba70a6c2 P d909237adf8645a190f96bbb1f9fcceb: Deleting consensus metadata
I20260924 19:31:40.124418 2101 catalog_manager.cc:4841] TS d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389): tablet 1cef495cbdd24d7c8ade4080ba70a6c2 (table test-table [id=4211a588f2cf4e519bffab75920a6431]) successfully deleted
I20260924 19:31:40.351712 2686 raft_consensus.cc:482] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:40.351814 2686 raft_consensus.cc:504] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.351977 2686 leader_election.cc:288] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363), edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:40.352191 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cf512f61b74834981b8e8d44ae4a07" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe786d7395b4d439c1484587c4f0448" is_pre_election: true
I20260924 19:31:40.352198 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cf512f61b74834981b8e8d44ae4a07" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb80b2c0f1e41608a9437427f8f08eb" is_pre_election: true
I20260924 19:31:40.352279 2412 raft_consensus.cc:2454] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 0.
I20260924 19:31:40.352284 2552 raft_consensus.cc:2454] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 0.
I20260924 19:31:40.352440 2203 leader_election.cc:302] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d909237adf8645a190f96bbb1f9fcceb, edb80b2c0f1e41608a9437427f8f08eb; no voters:
I20260924 19:31:40.352535 2686 raft_consensus.cc:2781] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:40.352619 2686 raft_consensus.cc:482] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:40.352666 2686 raft_consensus.cc:3037] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:40.353344 2686 raft_consensus.cc:504] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.353456 2686 leader_election.cc:288] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 election: Requested vote from peers 4fe786d7395b4d439c1484587c4f0448 (127.0.98.2:42363), edb80b2c0f1e41608a9437427f8f08eb (127.0.98.3:39133)
I20260924 19:31:40.353626 2412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cf512f61b74834981b8e8d44ae4a07" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fe786d7395b4d439c1484587c4f0448"
I20260924 19:31:40.353633 2552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6cf512f61b74834981b8e8d44ae4a07" candidate_uuid: "d909237adf8645a190f96bbb1f9fcceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb80b2c0f1e41608a9437427f8f08eb"
I20260924 19:31:40.353709 2412 raft_consensus.cc:3037] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:40.353739 2552 raft_consensus.cc:3037] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:40.354425 2412 raft_consensus.cc:2454] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 1.
I20260924 19:31:40.354425 2552 raft_consensus.cc:2454] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d909237adf8645a190f96bbb1f9fcceb in term 1.
I20260924 19:31:40.354589 2203 leader_election.cc:302] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d909237adf8645a190f96bbb1f9fcceb, edb80b2c0f1e41608a9437427f8f08eb; no voters:
I20260924 19:31:40.354705 2686 raft_consensus.cc:2781] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:40.354761 2686 raft_consensus.cc:686] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [term 1 LEADER]: Becoming Leader. State: Replica: d909237adf8645a190f96bbb1f9fcceb, State: Running, Role: LEADER
I20260924 19:31:40.354811 2686 consensus_queue.cc:237] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } }
I20260924 19:31:40.355352 2114 catalog_manager.cc:5495] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb reported cstate change: term changed from 0 to 1, leader changed from <none> to d909237adf8645a190f96bbb1f9fcceb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d909237adf8645a190f96bbb1f9fcceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d909237adf8645a190f96bbb1f9fcceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 } health_report { overall_health: UNKNOWN } } }
I20260924 19:31:40.407842 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:40.405903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.0.98.1:45389)
0924 19:31:40.405943 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:40.405945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:40.405954 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:40.405979 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:40.405979 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:40.405988 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:40.406029 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:40.406110 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.406585 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.407319 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:40.407506 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:40.407563 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:40.407640 (+ 77us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:40.407665 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:40.407694 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:40.407751 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 19:31:40.407767 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:40.409713 2552 raft_consensus.cc:1261] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Refusing update from remote peer d909237adf8645a190f96bbb1f9fcceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:40.409720 2412 raft_consensus.cc:1261] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Refusing update from remote peer d909237adf8645a190f96bbb1f9fcceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:40.409888 2686 consensus_queue.cc:1035] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fe786d7395b4d439c1484587c4f0448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:40.409989 2658 consensus_queue.cc:1035] T b6cf512f61b74834981b8e8d44ae4a07 P d909237adf8645a190f96bbb1f9fcceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb80b2c0f1e41608a9437427f8f08eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:31:40.413713 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:31:41.930418 2115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:41.930516 2115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:41.931313 2115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a14dcbbbd16c48f5ae3d06badd258729: Sending DeleteTablet for 3 replicas of tablet b6cf512f61b74834981b8e8d44ae4a07
I20260924 19:31:41.931563 2391 tablet_service.cc:1514] Processing DeleteTablet for tablet b6cf512f61b74834981b8e8d44ae4a07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44496
I20260924 19:31:41.931557 2532 tablet_service.cc:1514] Processing DeleteTablet for tablet b6cf512f61b74834981b8e8d44ae4a07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39026
I20260924 19:31:41.931677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 2187
I20260924 19:31:41.932798 2763 tablet_replica.cc:330] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: stopping tablet replica
I20260924 19:31:41.932853 2763 raft_consensus.cc:2229] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:41.932885 2763 raft_consensus.cc:2258] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:41.932973 2762 tablet_replica.cc:330] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448: stopping tablet replica
I20260924 19:31:41.933017 2762 raft_consensus.cc:2229] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:41.933046 2762 raft_consensus.cc:2258] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:41.933214 2763 ts_tablet_manager.cc:1905] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:41.934185 2762 ts_tablet_manager.cc:1905] T b6cf512f61b74834981b8e8d44ae4a07 P 4fe786d7395b4d439c1484587c4f0448: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:31:41.937739 2101 connection.cc:535] client connection to 127.0.98.1:45389 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:31:41.937852 2101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:31:41.938004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 2327
I20260924 19:31:41.938869 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:41.938659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.0.1:42878)
0924 19:31:41.938775 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:41.938827 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
W20260924 19:31:41.939345 2101 catalog_manager.cc:4584] TS d909237adf8645a190f96bbb1f9fcceb (127.0.98.1:45389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6cf512f61b74834981b8e8d44ae4a07: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45389: connect: Connection refused (error 111)
I20260924 19:31:41.992182 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:41.992027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.0.1:42880)
0924 19:31:41.992093 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:41.992140 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 19:31:41.992594 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:41.992473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45389 (local address 127.0.0.1:42882)
0924 19:31:41.992548 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:41.992574 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 19:31:42.012542 2763 ts_tablet_manager.cc:1918] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:31:42.012624 2763 log.cc:1196] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/b6cf512f61b74834981b8e8d44ae4a07
I20260924 19:31:42.012880 2763 ts_tablet_manager.cc:1939] T b6cf512f61b74834981b8e8d44ae4a07 P edb80b2c0f1e41608a9437427f8f08eb: Deleting consensus metadata
I20260924 19:31:42.017078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 2467
I20260924 19:31:42.017005 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:42.016889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42363 (local address 127.0.0.1:60460)
0924 19:31:42.016942 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:42.016965 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:31:42.021987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 2083
I20260924 19:31:42.022035 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:42.021931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39133 (local address 127.0.0.1:55090)
0924 19:31:42.021972 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:42.022000 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
2026-09-24T19:31:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5806 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:42 dist-test-slave-m8jb krb5kdc[2768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:42 dist-test-slave-m8jb krb5kdc[2768](info): set up 2 sockets
Sep 24 19:31:42 dist-test-slave-m8jb krb5kdc[2768](info): commencing operation
krb5kdc: starting...
W20260924 19:31:43.280229 392 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:43 dist-test-slave-m8jb krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:43.392105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42213
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:43.471066 2781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:43.471276 2781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:43.471297 2781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:43.472450 2781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:43.472491 2781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:43.472504 2781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:43.472517 2781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:43.472539 2781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:43.474131 2781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:43.474361 2781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:43.474576 2781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:31:43.479977 2781 server_base.cc:1034] running on GCE node
W20260924 19:31:43.479936 2786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:43.479928 2787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:43.479936 2789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:43.480541 2781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:43.480880 2781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:43.482023 2781 hybrid_clock.cc:648] HybridClock initialized: now 1790278303482011 us; error 31 us; skew 500 ppm
Sep 24 19:31:43 dist-test-slave-m8jb krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:43.484414 2781 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:43.484854 2781 webserver.cc:460] Webserver started at http://127.0.98.62:36381/ using document root <none> and password file <none>
I20260924 19:31:43.485064 2781 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:43.485105 2781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:43.485194 2781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:43.486240 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "2302e21986de413680c0858c537f4e3f"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "ce881cf66d1a6a8f6e2ccdb15736d947"
server_key_iv: "e135060abf6857e7af8031a34965e674"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.486539 2781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "2302e21986de413680c0858c537f4e3f"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "ce881cf66d1a6a8f6e2ccdb15736d947"
server_key_iv: "e135060abf6857e7af8031a34965e674"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.487753 2781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:43.488425 2796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:43.488590 2781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:43.488694 2781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "2302e21986de413680c0858c537f4e3f"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "ce881cf66d1a6a8f6e2ccdb15736d947"
server_key_iv: "e135060abf6857e7af8031a34965e674"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.488760 2781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:43.518051 2781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:43.520170 2781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:43.520282 2781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:43.523845 2781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42213
I20260924 19:31:43.523898 2848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42213 every 8 connection(s)
I20260924 19:31:43.524494 2781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:43.525187 2849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:43.526894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 2781
I20260924 19:31:43.526966 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:43.527115 392 external_mini_cluster.cc:1428] Setting key e4a236dc473040a54406e79b7d1cf36d
I20260924 19:31:43.527557 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f: Bootstrap starting.
I20260924 19:31:43.528298 2849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:43.528611 2849 log.cc:824] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:43.529349 2849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f: No bootstrap required, opened a new log
I20260924 19:31:43.531497 2849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2302e21986de413680c0858c537f4e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } }
I20260924 19:31:43.531666 2849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:43.531697 2849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2302e21986de413680c0858c537f4e3f, State: Initialized, Role: FOLLOWER
I20260924 19:31:43.531813 2849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2302e21986de413680c0858c537f4e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } }
I20260924 19:31:43.531879 2849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:43.531912 2849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:43.531957 2849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:43.532516 2849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2302e21986de413680c0858c537f4e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } }
I20260924 19:31:43.532603 2849 leader_election.cc:302] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2302e21986de413680c0858c537f4e3f; no voters:
I20260924 19:31:43.532789 2849 leader_election.cc:288] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:43.532841 2854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:43.533032 2849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:43.533041 2854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [term 1 LEADER]: Becoming Leader. State: Replica: 2302e21986de413680c0858c537f4e3f, State: Running, Role: LEADER
I20260924 19:31:43.533288 2854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2302e21986de413680c0858c537f4e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } }
I20260924 19:31:43.533684 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2302e21986de413680c0858c537f4e3f. Latest consensus state: current_term: 1 leader_uuid: "2302e21986de413680c0858c537f4e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2302e21986de413680c0858c537f4e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } } }
I20260924 19:31:43.533756 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:43.534106 2855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2302e21986de413680c0858c537f4e3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2302e21986de413680c0858c537f4e3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } } }
I20260924 19:31:43.534160 2855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f [sys.catalog]: This master's current role is: LEADER
Sep 24 19:31:43 dist-test-slave-m8jb krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260924 19:31:43.534905 2869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:31:43.534953 2869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:31:43.534996 2863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:43.535151 2863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:43.536708 2863 catalog_manager.cc:1347] Generated new cluster ID: 791c53671fe0419d97844dbbb4adb531
I20260924 19:31:43.536751 2863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:43.538175 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:43.527602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.0.98.62:42213)
0924 19:31:43.527784 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:43.527794 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:43.527824 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:31:43.528362 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:43.528372 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:43.528439 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:43.528625 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:43.529215 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.530570 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.531363 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.531971 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.532078 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:43.534676 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:43.534680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:43.534681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:43.534693 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:43.536531 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:43.536969 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:43.536972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:43.536973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:43.537079 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:43.537268 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:43.537269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:43.537269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:43.537572 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:43.537743 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:43.537828 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 19:31:43.537966 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260924 19:31:43.559723 2863 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:43.560285 2863 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:43.565804 2863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2302e21986de413680c0858c537f4e3f: Generated new TSK 0
I20260924 19:31:43.566030 2863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:43.596798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:43.705474 2875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:43.705694 2875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:43.705717 2875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:43.706838 2875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:43.706877 2875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:43.706902 2875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:43.708338 2875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:43.708523 2875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:43.708724 2875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:43.713312 2881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:43.713359 2880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:43.713349 2883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:43.713558 2875 server_base.cc:1034] running on GCE node
I20260924 19:31:43.713765 2875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:43.713972 2875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:43.715118 2875 hybrid_clock.cc:648] HybridClock initialized: now 1790278303715105 us; error 28 us; skew 500 ppm
Sep 24 19:31:43 dist-test-slave-m8jb krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:43.717759 2875 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:43.718173 2875 webserver.cc:460] Webserver started at http://127.0.98.1:33305/ using document root <none> and password file <none>
I20260924 19:31:43.718379 2875 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:43.718427 2875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:43.718557 2875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:43.719619 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "92340881c0d94ee7ac860ee9430a6007"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "768edb3578d0fb0518dd78ae4635c1ac"
server_key_iv: "a78b0e8b5d5af56afd7c4632228390af"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.719949 2875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "92340881c0d94ee7ac860ee9430a6007"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "768edb3578d0fb0518dd78ae4635c1ac"
server_key_iv: "a78b0e8b5d5af56afd7c4632228390af"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.721205 2875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 19:31:43.721931 2890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:43.722075 2875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:43.722155 2875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "92340881c0d94ee7ac860ee9430a6007"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "768edb3578d0fb0518dd78ae4635c1ac"
server_key_iv: "a78b0e8b5d5af56afd7c4632228390af"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.722213 2875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:43.729242 2875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:43.731107 2875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:43.731238 2875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:43.731439 2875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:43.731712 2875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:43.731745 2875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:43.731774 2875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:43.731791 2875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:43.737324 2875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37071
I20260924 19:31:43.737360 3003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37071 every 8 connection(s)
I20260924 19:31:43.737660 2875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:31:43.742041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 2875
I20260924 19:31:43.742122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:43.742274 392 external_mini_cluster.cc:1428] Setting key 5ca4f11f52fad12f32f752846c1feb86
Sep 24 19:31:43 dist-test-slave-m8jb krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:43.747084 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:43.738279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33061 (local address 127.0.98.62:42213)
0924 19:31:43.738340 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:43.738341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:43.738536 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:31:43.739501 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:43.739503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:43.739515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:43.739566 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:43.740162 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.740711 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.741669 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.741883 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.741936 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:43.744266 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:43.744267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:43.744268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:43.744276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:43.745793 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:43.746196 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:43.746197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:43.746198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:43.746257 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:43.746457 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:43.746457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:43.746458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:43.746594 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:43.746659 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:43.746899 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:31:43.746988 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:31:43.747164 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:43.738265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42213 (local address 127.0.98.1:33061)
0924 19:31:43.738418 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:43.738454 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:43.739395 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:43.739640 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:43.739644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:43.739886 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:43.740070 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:43.740074 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.740799 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.740800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:43.741589 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:43.741591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.741951 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.741952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:43.742079 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:43.742598 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:43.742608 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:43.744175 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:43.745881 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:43.745887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:43.745889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:43.746115 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:43.746314 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:43.746314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:43.746315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:43.746407 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:43.746657 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:43.746661 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:43.746778 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 19:31:43.746859 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:31:43.746984 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260924 19:31:43.747593 3004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42213
I20260924 19:31:43.747712 3004 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:43.747880 3004 heartbeater.cc:511] Master 127.0.98.62:42213 requested a full tablet report, sending...
I20260924 19:31:43.748353 2813 ts_manager.cc:194] Registered new tserver with Master: 92340881c0d94ee7ac860ee9430a6007 (127.0.98.1:37071)
I20260924 19:31:43.748996 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33061
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:43.767527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:43.847702 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:43.847929 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:43.847961 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:43.849162 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:43.849200 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:43.849236 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:43.850718 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:43.850950 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:43.851161 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:43.856237 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:43.856246 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:43.856304 3009 server_base.cc:1034] running on GCE node
W20260924 19:31:43.856261 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:43.856747 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:43.856998 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:43.858151 3009 hybrid_clock.cc:648] HybridClock initialized: now 1790278303858137 us; error 29 us; skew 500 ppm
Sep 24 19:31:43 dist-test-slave-m8jb krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:43.860710 3009 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:43.861202 3009 webserver.cc:460] Webserver started at http://127.0.98.2:36423/ using document root <none> and password file <none>
I20260924 19:31:43.861429 3009 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:43.861482 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:43.861706 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:43.862805 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "ab77a66b54864a14a6b9de2ec12dbfb7"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "c6f337f29df1c60db1bc21172f0c3211"
server_key_iv: "e48ce80683aa471a17a49e97c61436fc"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.863174 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ab77a66b54864a14a6b9de2ec12dbfb7"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "c6f337f29df1c60db1bc21172f0c3211"
server_key_iv: "e48ce80683aa471a17a49e97c61436fc"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.864602 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:43.865360 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:43.865556 3009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:43.865648 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "ab77a66b54864a14a6b9de2ec12dbfb7"
format_stamp: "Formatted at 2026-09-24 19:31:43 on dist-test-slave-m8jb"
server_key: "c6f337f29df1c60db1bc21172f0c3211"
server_key_iv: "e48ce80683aa471a17a49e97c61436fc"
server_key_version: "encryptionkey@0"
I20260924 19:31:43.865732 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:43.896728 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:43.898855 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:43.899006 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:43.899237 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:43.899552 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:43.899590 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:43.899624 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:43.899648 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:43.905704 3009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33323
I20260924 19:31:43.905766 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33323 every 8 connection(s)
I20260924 19:31:43.906069 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:31:43 dist-test-slave-m8jb krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:43.912427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 3009
I20260924 19:31:43.912515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:43.912732 392 external_mini_cluster.cc:1428] Setting key ecd91dd8b7dbec279b960b3d0526183b
I20260924 19:31:43.915455 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:43.906743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48191 (local address 127.0.98.62:42213)
0924 19:31:43.906786 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:43.906789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:43.906983 (+ 194us) server_negotiation.cc:407] Connection header received
0924 19:31:43.907949 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:43.907951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:43.907963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:43.908033 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:43.908703 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.909205 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.910234 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.910416 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.910471 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:43.912680 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:43.912682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:43.912682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:43.912690 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:43.914150 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:43.914591 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:43.914592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:43.914593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:43.914661 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:43.914883 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:43.914883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:43.914884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:43.915007 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:43.915079 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:43.915287 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 19:31:43.915364 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:31:43.915539 3140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:43.906718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42213 (local address 127.0.98.2:48191)
0924 19:31:43.906872 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:43.906914 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:43.907813 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:43.908126 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:43.908131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:43.908385 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:43.908572 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:43.908576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.909298 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.909299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:43.910153 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:43.910155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:43.910503 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:43.910504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:43.910627 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:43.911117 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:43.911125 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:43.912526 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:43.914226 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:43.914233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:43.914235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:43.914506 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:43.914727 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:43.914728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:43.914728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:43.914833 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:43.915058 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:43.915061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:43.915172 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 19:31:43.915254 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:31:43.915371 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260924 19:31:43.915918 3138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42213
I20260924 19:31:43.916016 3138 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:43.916188 3138 heartbeater.cc:511] Master 127.0.98.62:42213 requested a full tablet report, sending...
I20260924 19:31:43.916574 2813 ts_manager.cc:194] Registered new tserver with Master: ab77a66b54864a14a6b9de2ec12dbfb7 (127.0.98.2:33323)
I20260924 19:31:43.916960 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48191
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:43.937625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:44.016839 3143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:44.017046 3143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:44.017068 3143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:44.018252 3143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:44.018296 3143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:44.018321 3143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:44.019842 3143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:44.020066 3143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:44.020292 3143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:44.025188 3148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:44.025177 3151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:44.025290 3149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:44.025300 3143 server_base.cc:1034] running on GCE node
I20260924 19:31:44.025578 3143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:44.025776 3143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:44.026907 3143 hybrid_clock.cc:648] HybridClock initialized: now 1790278304026892 us; error 30 us; skew 500 ppm
Sep 24 19:31:44 dist-test-slave-m8jb krb5kdc[2768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:44.029332 3143 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:44.029722 3143 webserver.cc:460] Webserver started at http://127.0.98.3:39135/ using document root <none> and password file <none>
I20260924 19:31:44.029927 3143 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:44.029974 3143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:44.030094 3143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:44.031077 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "2180a52a4f9b4a84b845db9597860a8c"
format_stamp: "Formatted at 2026-09-24 19:31:44 on dist-test-slave-m8jb"
server_key: "376e50a59aeda6b27c90accf90aca55a"
server_key_iv: "8d78308416b099d6d9cffa55d3b1027d"
server_key_version: "encryptionkey@0"
I20260924 19:31:44.031396 3143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2180a52a4f9b4a84b845db9597860a8c"
format_stamp: "Formatted at 2026-09-24 19:31:44 on dist-test-slave-m8jb"
server_key: "376e50a59aeda6b27c90accf90aca55a"
server_key_iv: "8d78308416b099d6d9cffa55d3b1027d"
server_key_version: "encryptionkey@0"
I20260924 19:31:44.032579 3143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:31:44.033278 3158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:44.033452 3143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:44.033519 3143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "2180a52a4f9b4a84b845db9597860a8c"
format_stamp: "Formatted at 2026-09-24 19:31:44 on dist-test-slave-m8jb"
server_key: "376e50a59aeda6b27c90accf90aca55a"
server_key_iv: "8d78308416b099d6d9cffa55d3b1027d"
server_key_version: "encryptionkey@0"
I20260924 19:31:44.033579 3143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:44.053357 3143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:44.055398 3143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:44.055526 3143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:44.055720 3143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:44.055977 3143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:44.056012 3143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:44.056042 3143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:44.056062 3143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:44.062203 3143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42585
I20260924 19:31:44.062273 3271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42585 every 8 connection(s)
I20260924 19:31:44.062579 3143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:31:44 dist-test-slave-m8jb krb5kdc[2768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:44.071457 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:44.063305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36123 (local address 127.0.98.62:42213)
0924 19:31:44.063367 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:44.063369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:44.063543 (+ 174us) server_negotiation.cc:407] Connection header received
0924 19:31:44.064473 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:44.064474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:44.064484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:44.064536 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:44.065182 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:44.065743 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:44.066849 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:44.067039 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:44.067095 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:44.069182 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:44.069183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:44.069183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:44.069191 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:44.070459 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:44.070785 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:44.070785 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:44.070786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:44.070829 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:44.070995 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:44.070996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:44.070996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:44.071085 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:44.071127 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:44.071337 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 19:31:44.071392 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:31:44.071677 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:44.063266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42213 (local address 127.0.98.3:36123)
0924 19:31:44.063414 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:44.063455 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:44.064363 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:44.064600 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:44.064603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:44.064900 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:44.065102 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:44.065106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:44.065845 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:44.065846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:44.066768 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:44.066771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:44.067129 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:44.067130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:44.067257 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:44.067691 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:44.067699 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:44.069100 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:44.070523 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:44.070529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:44.070530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:44.070732 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:44.070871 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:44.070871 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:44.070872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:44.070955 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:44.071124 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:44.071127 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:44.071234 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:31:44.071318 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:31:44.071434 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260924 19:31:44.072109 3272 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42213
I20260924 19:31:44.072199 3272 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:44.072320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 3143
I20260924 19:31:44.072368 3272 heartbeater.cc:511] Master 127.0.98.62:42213 requested a full tablet report, sending...
I20260924 19:31:44.072392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:44.072567 392 external_mini_cluster.cc:1428] Setting key 1d447a8fb0c78c9856ba86e5ba868f70
I20260924 19:31:44.072753 2813 ts_manager.cc:194] Registered new tserver with Master: 2180a52a4f9b4a84b845db9597860a8c (127.0.98.3:42585)
I20260924 19:31:44.073163 2813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36123
I20260924 19:31:44.073396 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:31:44.084064 2852 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:31:44.083082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.0.98.62:42213)
0924 19:31:44.083164 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:44.083166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:44.083193 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:31:44.083261 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:44.083262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:44.083271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:44.083320 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:44.083987 (+ 667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35170: BlockingRecv error: recv got EOF from 127.0.0.1:35170 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:31:44.084946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 2875
I20260924 19:31:44.089294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 3009
I20260924 19:31:44.092864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 3143
I20260924 19:31:44.096196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 2781
2026-09-24T19:31:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2071 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:44 dist-test-slave-m8jb krb5kdc[3284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:44 dist-test-slave-m8jb krb5kdc[3284](info): set up 2 sockets
Sep 24 19:31:44 dist-test-slave-m8jb krb5kdc[3284](info): commencing operation
krb5kdc: starting...
W20260924 19:31:45.040162 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:45 dist-test-slave-m8jb krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:45.155613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35125
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:45.232581 3297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:45.232826 3297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:45.232856 3297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:45.233987 3297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:45.234025 3297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:45.234043 3297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:45.234058 3297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:45.234072 3297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:45.235477 3297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:45.235752 3297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:45.235948 3297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:31:45.240739 3297 server_base.cc:1034] running on GCE node
W20260924 19:31:45.240720 3305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:45.240700 3303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:45.240720 3302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:45.241096 3297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:45.241369 3297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:45.242537 3297 hybrid_clock.cc:648] HybridClock initialized: now 1790278305242519 us; error 28 us; skew 500 ppm
Sep 24 19:31:45 dist-test-slave-m8jb krb5kdc[3284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:45.244928 3297 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:45.245330 3297 webserver.cc:460] Webserver started at http://127.0.98.62:36611/ using document root <none> and password file <none>
I20260924 19:31:45.245518 3297 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:45.245565 3297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:45.245702 3297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:45.246796 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "13f5926c86e24c669f5140fda1c822cf"
format_stamp: "Formatted at 2026-09-24 19:31:45 on dist-test-slave-m8jb"
server_key: "cd1050c4af03ea1df43192a393f01804"
server_key_iv: "77bca8acda93304097ad6430cc84e41d"
server_key_version: "encryptionkey@0"
I20260924 19:31:45.247119 3297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "13f5926c86e24c669f5140fda1c822cf"
format_stamp: "Formatted at 2026-09-24 19:31:45 on dist-test-slave-m8jb"
server_key: "cd1050c4af03ea1df43192a393f01804"
server_key_iv: "77bca8acda93304097ad6430cc84e41d"
server_key_version: "encryptionkey@0"
I20260924 19:31:45.248390 3297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:31:45.249264 3312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:45.249411 3297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:45.249498 3297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "13f5926c86e24c669f5140fda1c822cf"
format_stamp: "Formatted at 2026-09-24 19:31:45 on dist-test-slave-m8jb"
server_key: "cd1050c4af03ea1df43192a393f01804"
server_key_iv: "77bca8acda93304097ad6430cc84e41d"
server_key_version: "encryptionkey@0"
I20260924 19:31:45.249567 3297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:45.280872 3297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:45.283027 3297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:45.283165 3297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:45.286880 3297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35125
I20260924 19:31:45.286927 3364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35125 every 8 connection(s)
I20260924 19:31:45.287261 3297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:45.288120 3365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:45.289980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 3297
I20260924 19:31:45.290089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:45.290247 392 external_mini_cluster.cc:1428] Setting key e73a7aee8529c037de1bb889b9da322e
I20260924 19:31:45.290512 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf: Bootstrap starting.
I20260924 19:31:45.291155 3365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:45.291427 3365 log.cc:824] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:45.292049 3365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf: No bootstrap required, opened a new log
I20260924 19:31:45.294044 3365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f5926c86e24c669f5140fda1c822cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260924 19:31:45.294196 3365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:45.294231 3365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f5926c86e24c669f5140fda1c822cf, State: Initialized, Role: FOLLOWER
I20260924 19:31:45.294360 3365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f5926c86e24c669f5140fda1c822cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260924 19:31:45.294440 3365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:45.294480 3365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:45.294526 3365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:45.295106 3365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f5926c86e24c669f5140fda1c822cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260924 19:31:45.295215 3365 leader_election.cc:302] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13f5926c86e24c669f5140fda1c822cf; no voters:
I20260924 19:31:45.295373 3365 leader_election.cc:288] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:45.295408 3370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:45.295529 3370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [term 1 LEADER]: Becoming Leader. State: Replica: 13f5926c86e24c669f5140fda1c822cf, State: Running, Role: LEADER
I20260924 19:31:45.295645 3365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:45.295688 3370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f5926c86e24c669f5140fda1c822cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } }
I20260924 19:31:45.296037 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13f5926c86e24c669f5140fda1c822cf. Latest consensus state: current_term: 1 leader_uuid: "13f5926c86e24c669f5140fda1c822cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f5926c86e24c669f5140fda1c822cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } } }
I20260924 19:31:45.296123 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:45.296365 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13f5926c86e24c669f5140fda1c822cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f5926c86e24c669f5140fda1c822cf" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35125 } } }
I20260924 19:31:45.296495 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf [sys.catalog]: This master's current role is: LEADER
Sep 24 19:31:45 dist-test-slave-m8jb krb5kdc[3284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260924 19:31:45.297324 3386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:31:45.297377 3386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:31:45.297417 3378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:45.297549 3378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:45.299031 3378 catalog_manager.cc:1347] Generated new cluster ID: a80c3a9ac5bc41f2965105fa2427dd2a
I20260924 19:31:45.299091 3378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:45.300869 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:45.290721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.0.98.62:35125)
0924 19:31:45.290919 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:45.290929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:45.290960 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:31:45.291443 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:45.291451 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:45.291515 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:45.291665 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:45.292352 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:45.293286 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:45.294059 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:45.294566 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:45.294676 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:45.297573 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:45.297578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:45.297578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:45.297591 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:45.299352 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:45.299710 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:45.299713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:45.299713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:45.299802 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:45.299989 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:45.299990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:45.299990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:45.300231 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:45.300388 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:45.300508 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:31:45.300670 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260924 19:31:45.305179 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:31:45.314404 3378 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:45.315130 3378 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:45.319317 3378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13f5926c86e24c669f5140fda1c822cf: Generated new TSK 0
I20260924 19:31:45.319510 3378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:31:45.322661 3368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:45.316688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.0.98.62:35125)
0924 19:31:45.316746 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:45.316749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:45.319069 (+ 2320us) server_negotiation.cc:407] Connection header received
0924 19:31:45.319144 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:45.319145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:45.319157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:45.319220 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:45.319708 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:45.320510 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:45.321254 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:45.321567 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:45.321642 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:45.322211 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:45.322212 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:45.322213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:45.322432 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:45.322450 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:45.322457 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:45.322512 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:45.322559 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:31:45.322583 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:31:45.324847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 3297
2026-09-24T19:31:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1226 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:45 dist-test-slave-m8jb krb5kdc[3398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:45 dist-test-slave-m8jb krb5kdc[3398](info): set up 2 sockets
Sep 24 19:31:45 dist-test-slave-m8jb krb5kdc[3398](info): commencing operation
krb5kdc: starting...
W20260924 19:31:46.264278 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:46.377924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44649
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:46.455539 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:46.455752 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:46.455781 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:46.456985 3411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:46.457031 3411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:46.457047 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:46.457064 3411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:46.457079 3411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:46.458621 3411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:46.458887 3411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:46.459103 3411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:46.463761 3419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:46.463814 3417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:46.463953 3411 server_base.cc:1034] running on GCE node
W20260924 19:31:46.463814 3416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:46.464236 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:46.464504 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:46.465638 3411 hybrid_clock.cc:648] HybridClock initialized: now 1790278306465628 us; error 26 us; skew 500 ppm
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:46.474323 3411 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:46.474776 3411 webserver.cc:460] Webserver started at http://127.0.98.62:37893/ using document root <none> and password file <none>
I20260924 19:31:46.474978 3411 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:46.475018 3411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:46.475139 3411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:46.476198 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "8044595570da7fc2870124d572fea1bc"
server_key_iv: "d543707d5070634f713ab4f2db5351b3"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.476511 3411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "8044595570da7fc2870124d572fea1bc"
server_key_iv: "d543707d5070634f713ab4f2db5351b3"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.477828 3411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:46.478502 3426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.478660 3411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:46.478741 3411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "8044595570da7fc2870124d572fea1bc"
server_key_iv: "d543707d5070634f713ab4f2db5351b3"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.478809 3411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:46.494660 3411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:46.496829 3411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:46.496965 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:46.500675 3411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44649
I20260924 19:31:46.500710 3478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44649 every 8 connection(s)
I20260924 19:31:46.501044 3411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:46.502036 3479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:46.502342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 3411
I20260924 19:31:46.502441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:46.502586 392 external_mini_cluster.cc:1428] Setting key aa6e737f5af055e8ad2b0eff58d48b96
I20260924 19:31:46.504402 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1: Bootstrap starting.
I20260924 19:31:46.505023 3479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:46.505285 3479 log.cc:824] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:46.505959 3479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1: No bootstrap required, opened a new log
I20260924 19:31:46.508113 3479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44649 } }
I20260924 19:31:46.508275 3479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:46.508319 3479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096eb9ffa7684c36a3d1eeb59e6a3cf1, State: Initialized, Role: FOLLOWER
I20260924 19:31:46.508436 3479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44649 } }
I20260924 19:31:46.508517 3479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:46.508550 3479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:46.508591 3479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:46.509249 3479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44649 } }
I20260924 19:31:46.509361 3479 leader_election.cc:302] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 096eb9ffa7684c36a3d1eeb59e6a3cf1; no voters:
I20260924 19:31:46.509542 3479 leader_election.cc:288] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:46.509596 3484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:46.509745 3484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [term 1 LEADER]: Becoming Leader. State: Replica: 096eb9ffa7684c36a3d1eeb59e6a3cf1, State: Running, Role: LEADER
I20260924 19:31:46.509811 3479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:46.509851 3484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44649 } }
I20260924 19:31:46.510183 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 096eb9ffa7684c36a3d1eeb59e6a3cf1. Latest consensus state: current_term: 1 leader_uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44649 } } }
I20260924 19:31:46.510251 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:46.510366 3485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096eb9ffa7684c36a3d1eeb59e6a3cf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44649 } } }
I20260924 19:31:46.510444 3485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:46.510673 3490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:46.511127 3490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:46.512741 3490 catalog_manager.cc:1347] Generated new cluster ID: f6e4890770684742b1ab5dfe4ec361a1
I20260924 19:31:46.512796 3490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:46.515103 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:46.503073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.0.98.62:44649)
0924 19:31:46.503245 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:46.503253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:46.504744 (+ 1491us) server_negotiation.cc:407] Connection header received
0924 19:31:46.505243 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:46.505250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:46.505305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:46.505437 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:46.506105 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.507445 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.508203 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.508823 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.508934 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:46.511666 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:46.511670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:46.511670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:46.511680 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:46.513544 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:46.513959 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:46.513962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:46.513962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:46.514065 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:46.514254 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:46.514255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:46.514255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:46.514528 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:46.514658 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:46.514782 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:31:46.514905 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260924 19:31:46.522665 3490 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:46.523393 3490 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:46.526851 3490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 096eb9ffa7684c36a3d1eeb59e6a3cf1: Generated new TSK 0
I20260924 19:31:46.527014 3490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:46.553784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44649
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:46.633143 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:46.633342 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:46.633366 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:46.634516 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:46.634552 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:46.634579 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:46.636011 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:46.636221 3505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:46.636410 3505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:46.641356 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:46.641373 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:46.641362 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:46.641498 3505 server_base.cc:1034] running on GCE node
I20260924 19:31:46.641721 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:46.641986 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:46.643136 3505 hybrid_clock.cc:648] HybridClock initialized: now 1790278306643118 us; error 33 us; skew 500 ppm
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:46.645756 3505 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:46.646204 3505 webserver.cc:460] Webserver started at http://127.0.98.1:36405/ using document root <none> and password file <none>
I20260924 19:31:46.646426 3505 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:46.646474 3505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:46.646597 3505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:46.647675 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "93eb292af6cd45f8b87a55415faf4e55"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "08697cb9aa05fcf7489a0f8ec228718d"
server_key_iv: "4092c55743ff78573c4860be70a37aed"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.648032 3505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93eb292af6cd45f8b87a55415faf4e55"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "08697cb9aa05fcf7489a0f8ec228718d"
server_key_iv: "4092c55743ff78573c4860be70a37aed"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.649431 3505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:46.650198 3520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.650404 3505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:46.650491 3505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "93eb292af6cd45f8b87a55415faf4e55"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "08697cb9aa05fcf7489a0f8ec228718d"
server_key_iv: "4092c55743ff78573c4860be70a37aed"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.650552 3505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:46.659543 3505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:46.661633 3505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:46.661758 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:46.661957 3505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:46.662230 3505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:46.662271 3505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.662299 3505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:46.662323 3505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.667673 3505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34023
I20260924 19:31:46.667714 3633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34023 every 8 connection(s)
I20260924 19:31:46.668035 3505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:46.676225 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:46.668734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58071 (local address 127.0.98.62:44649)
0924 19:31:46.668793 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:46.668795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:46.668977 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:31:46.669428 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:46.669429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:46.669439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:46.669502 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:46.670376 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.670873 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.671820 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.672020 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.672072 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:46.674099 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:46.674100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:46.674101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:46.674107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:46.675199 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:46.675536 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:46.675537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:46.675537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:46.675580 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:46.675734 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:46.675735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:46.675735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:46.675841 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:46.675886 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:46.676091 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:31:46.676154 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:31:46.676420 3636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:46.668682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44649 (local address 127.0.98.1:58071)
0924 19:31:46.668834 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:46.668875 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:46.669354 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:46.669555 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:46.669558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:46.670092 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:46.670295 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:46.670301 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.670936 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.670936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:46.671726 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:46.671728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.672080 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.672080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:46.672170 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:46.672660 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:46.672667 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:46.674020 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:46.675275 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:46.675281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:46.675282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:46.675485 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:46.675616 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:46.675617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:46.675617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:46.675697 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:46.675884 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:46.675887 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:46.675999 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:31:46.676076 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:31:46.676196 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260924 19:31:46.676925 3634 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44649
I20260924 19:31:46.677007 3634 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:46.677152 3634 heartbeater.cc:511] Master 127.0.98.62:44649 requested a full tablet report, sending...
I20260924 19:31:46.677594 3443 ts_manager.cc:194] Registered new tserver with Master: 93eb292af6cd45f8b87a55415faf4e55 (127.0.98.1:34023)
I20260924 19:31:46.678145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 3505
I20260924 19:31:46.678149 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58071
I20260924 19:31:46.678226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:46.678419 392 external_mini_cluster.cc:1428] Setting key 22435693802fd6dd62b025a4e8025ba7
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:46.700968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44649
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:46.779879 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:46.780093 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:46.780128 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:46.781316 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:46.781358 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:46.781391 3639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:46.782846 3639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:46.783090 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:46.783298 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:46.787916 3645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:46.787925 3644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:46.787948 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:46.788115 3639 server_base.cc:1034] running on GCE node
I20260924 19:31:46.788317 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:46.788576 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:46.789742 3639 hybrid_clock.cc:648] HybridClock initialized: now 1790278306789737 us; error 30 us; skew 500 ppm
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:46.792050 3639 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:46.792485 3639 webserver.cc:460] Webserver started at http://127.0.98.2:45601/ using document root <none> and password file <none>
I20260924 19:31:46.792726 3639 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:46.792774 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:46.792976 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:46.794027 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "d3a368099c8f41fa83422438bb89d163"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "06e6e91cbff2e28062bbda473ccbcc88"
server_key_iv: "0026aaec15863baf6d929a255666a34f"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.794355 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d3a368099c8f41fa83422438bb89d163"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "06e6e91cbff2e28062bbda473ccbcc88"
server_key_iv: "0026aaec15863baf6d929a255666a34f"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.795535 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:31:46.796172 3654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.796301 3639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:46.796376 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "d3a368099c8f41fa83422438bb89d163"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "06e6e91cbff2e28062bbda473ccbcc88"
server_key_iv: "0026aaec15863baf6d929a255666a34f"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.796440 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:46.810024 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:46.812211 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:46.812345 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:46.812568 3639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:46.812927 3639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:46.812963 3639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.812992 3639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:46.813009 3639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.818311 3639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38873
I20260924 19:31:46.818358 3767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38873 every 8 connection(s)
I20260924 19:31:46.818660 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:46.825304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 3639
I20260924 19:31:46.825387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:46.825587 392 external_mini_cluster.cc:1428] Setting key 2cccc33695d8c8aa4891f06d16e1e6a2
I20260924 19:31:46.827373 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:46.819292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53033 (local address 127.0.98.62:44649)
0924 19:31:46.819350 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:46.819352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:46.819546 (+ 194us) server_negotiation.cc:407] Connection header received
0924 19:31:46.820009 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:46.820010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:46.820019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:46.820072 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:46.821013 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.821502 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.822493 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.822668 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.822711 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:46.824620 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:46.824622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:46.824622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:46.824647 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:46.826062 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:46.826474 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:46.826475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:46.826476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:46.826531 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:46.826805 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:46.826805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:46.826806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:46.826906 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:46.826949 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:46.827241 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:31:46.827308 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:31:46.827634 3770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:46.819277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44649 (local address 127.0.98.2:53033)
0924 19:31:46.819426 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:46.819467 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:46.819935 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:46.820142 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:46.820145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:46.820747 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:46.820941 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:46.820947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.821597 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.821598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:46.822429 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:46.822431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.822722 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.822722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:46.822815 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:46.823215 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:46.823223 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:46.824539 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:46.826130 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:46.826137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:46.826139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:46.826406 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:46.826586 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:46.826587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:46.826587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:46.826745 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:46.826953 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:46.826958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:46.827120 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 19:31:46.827227 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:31:46.827393 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260924 19:31:46.828121 3768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44649
I20260924 19:31:46.828204 3768 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:46.828367 3768 heartbeater.cc:511] Master 127.0.98.62:44649 requested a full tablet report, sending...
I20260924 19:31:46.828737 3443 ts_manager.cc:194] Registered new tserver with Master: d3a368099c8f41fa83422438bb89d163 (127.0.98.2:38873)
I20260924 19:31:46.829200 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53033
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:46.849696 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44649
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:46.927740 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:46.928053 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:46.928079 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:46.929436 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:46.929482 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:46.929508 3773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:46.930907 3773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44649
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:46.931084 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:46.931257 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:46.935786 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:46.935906 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:46.936007 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:46.936038 3773 server_base.cc:1034] running on GCE node
I20260924 19:31:46.936199 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:46.936376 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:46.937511 3773 hybrid_clock.cc:648] HybridClock initialized: now 1790278306937496 us; error 27 us; skew 500 ppm
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:46.939831 3773 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:46.940241 3773 webserver.cc:460] Webserver started at http://127.0.98.3:42781/ using document root <none> and password file <none>
I20260924 19:31:46.940407 3773 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:46.940444 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:46.940529 3773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:46.941542 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d0a625ccdf4450dab1f0fa96b90e374"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "c299b350cae415d6c8d688703c658c65"
server_key_iv: "6e05c8a145bfb67219f392efa3e6c923"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.941833 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d0a625ccdf4450dab1f0fa96b90e374"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "c299b350cae415d6c8d688703c658c65"
server_key_iv: "6e05c8a145bfb67219f392efa3e6c923"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.942961 3773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:46.943596 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.943780 3773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:46.943859 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "9d0a625ccdf4450dab1f0fa96b90e374"
format_stamp: "Formatted at 2026-09-24 19:31:46 on dist-test-slave-m8jb"
server_key: "c299b350cae415d6c8d688703c658c65"
server_key_iv: "6e05c8a145bfb67219f392efa3e6c923"
server_key_version: "encryptionkey@0"
I20260924 19:31:46.943920 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:46.970453 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:46.972265 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:46.972404 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:46.972693 3773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:46.973044 3773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:46.973088 3773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.973119 3773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:46.973135 3773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:46.978622 3773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33077
I20260924 19:31:46.978677 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33077 every 8 connection(s)
I20260924 19:31:46.978976 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:31:46.984197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 3773
I20260924 19:31:46.984301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:46.984483 392 external_mini_cluster.cc:1428] Setting key e8b3997ae0ce3ffce2fca25a164fa64f
Sep 24 19:31:46 dist-test-slave-m8jb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:46.987552 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:46.979649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56817 (local address 127.0.98.62:44649)
0924 19:31:46.979690 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:46.979693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:46.979978 (+ 285us) server_negotiation.cc:407] Connection header received
0924 19:31:46.980380 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:46.980381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:46.980391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:46.980447 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:46.981412 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.981940 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.982938 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.983135 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.983188 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:46.985303 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:46.985304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:46.985304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:46.985312 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:46.986576 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:46.986893 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:46.986894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:46.986894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:46.986941 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:46.987097 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:46.987097 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:46.987097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:46.987190 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:46.987233 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:46.987432 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 19:31:46.987489 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:31:46.987757 3904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:46.979631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44649 (local address 127.0.98.3:56817)
0924 19:31:46.979779 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:46.979821 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:46.980287 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:46.980503 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:46.980506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:46.981097 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:46.981328 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:46.981333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.982017 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.982018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:46.982856 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:46.982858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:46.983198 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:46.983198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:46.983294 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:46.983709 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:46.983718 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:46.985229 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:46.986640 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:46.986644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:46.986645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:46.986840 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:46.986977 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:46.986978 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:46.986978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:46.987058 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:46.987232 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:46.987235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:46.987345 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 19:31:46.987421 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:31:46.987534 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260924 19:31:46.988166 3902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44649
I20260924 19:31:46.988242 3902 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:46.988401 3902 heartbeater.cc:511] Master 127.0.98.62:44649 requested a full tablet report, sending...
I20260924 19:31:46.988739 3443 ts_manager.cc:194] Registered new tserver with Master: 9d0a625ccdf4450dab1f0fa96b90e374 (127.0.98.3:33077)
I20260924 19:31:46.989109 3443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56817
I20260924 19:31:46.996104 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:31:47 dist-test-slave-m8jb krb5kdc[3398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 19:31:47 dist-test-slave-m8jb krb5kdc[3398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:47.017740 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:47.010712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.0.98.62:44649)
0924 19:31:47.010780 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:47.010782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:47.010883 (+ 101us) server_negotiation.cc:407] Connection header received
0924 19:31:47.010948 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:47.010949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:47.010959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:47.011003 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:47.011645 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:47.012139 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:47.012923 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:47.013106 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:47.013158 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:47.015516 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:47.015517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:47.015518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:47.015525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:47.016864 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:47.017145 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:47.017146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:47.017146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:47.017193 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:47.017317 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:47.017318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:47.017318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:47.017456 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:47.017528 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:47.017603 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 19:31:47.017663 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 19:31:47.018014 3443 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50834
I20260924 19:31:47.018478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 3505
I20260924 19:31:47.022488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 3639
I20260924 19:31:47.026155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 3773
I20260924 19:31:47.029723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 3411
2026-09-24T19:31:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1706 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:47 dist-test-slave-m8jb krb5kdc[3914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:47 dist-test-slave-m8jb krb5kdc[3914](info): set up 2 sockets
Sep 24 19:31:47 dist-test-slave-m8jb krb5kdc[3914](info): commencing operation
krb5kdc: starting...
W20260924 19:31:48.033241 392 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 19:31:48.235152 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.202s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 19:31:48.429045 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.194s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 19:31:48.609092 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.180s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/test-user.keytab.
W20260924 19:31:49.160223 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.551s user 0.001s sys 0.001s
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:49.201644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:49.280140 3927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:49.280347 3927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:49.280367 3927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:49.281563 3927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:49.281615 3927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:49.281627 3927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:49.281638 3927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:49.281651 3927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:49.283077 3927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:49.283308 3927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:49.283512 3927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:49.288084 3933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:49.288085 3935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:49.288168 3932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:49.288357 3927 server_base.cc:1034] running on GCE node
I20260924 19:31:49.288620 3927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:49.288945 3927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:49.290122 3927 hybrid_clock.cc:648] HybridClock initialized: now 1790278309290107 us; error 31 us; skew 500 ppm
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:49.292496 3927 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:49.292982 3927 webserver.cc:460] Webserver started at http://127.0.98.62:35899/ using document root <none> and password file <none>
I20260924 19:31:49.293183 3927 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:49.293227 3927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:49.293354 3927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:49.294394 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "bb897562d0de40109d5696f1b3b85905"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "642c0b2e7486a407af493962027bcda0"
server_key_iv: "d0320b7437850300dd825ddd930d1843"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.294710 3927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb897562d0de40109d5696f1b3b85905"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "642c0b2e7486a407af493962027bcda0"
server_key_iv: "d0320b7437850300dd825ddd930d1843"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.295948 3927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:31:49.296595 3942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.296782 3927 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:49.296857 3927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "bb897562d0de40109d5696f1b3b85905"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "642c0b2e7486a407af493962027bcda0"
server_key_iv: "d0320b7437850300dd825ddd930d1843"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.296926 3927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:49.314275 3927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:49.316301 3927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:49.316426 3927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:49.320006 3927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34859
I20260924 19:31:49.320070 3994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34859 every 8 connection(s)
I20260924 19:31:49.320365 3927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:49.321197 3995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:49.323359 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905: Bootstrap starting.
I20260924 19:31:49.323860 3995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:49.324085 3995 log.cc:824] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:49.324716 3995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905: No bootstrap required, opened a new log
I20260924 19:31:49.326083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 3927
I20260924 19:31:49.326195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:49.326371 392 external_mini_cluster.cc:1428] Setting key 4e0621045eac8e2d856313482851e78a
I20260924 19:31:49.326421 3995 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb897562d0de40109d5696f1b3b85905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260924 19:31:49.326582 3995 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:49.326617 3995 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb897562d0de40109d5696f1b3b85905, State: Initialized, Role: FOLLOWER
I20260924 19:31:49.326751 3995 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb897562d0de40109d5696f1b3b85905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260924 19:31:49.326854 3995 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:49.326921 3995 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:49.326978 3995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:49.327541 3995 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb897562d0de40109d5696f1b3b85905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260924 19:31:49.327623 3995 leader_election.cc:302] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb897562d0de40109d5696f1b3b85905; no voters:
I20260924 19:31:49.327769 3995 leader_election.cc:288] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:49.327821 4000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:49.327997 4000 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [term 1 LEADER]: Becoming Leader. State: Replica: bb897562d0de40109d5696f1b3b85905, State: Running, Role: LEADER
I20260924 19:31:49.328011 3995 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:49.328173 4000 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb897562d0de40109d5696f1b3b85905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } }
I20260924 19:31:49.329381 4001 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb897562d0de40109d5696f1b3b85905. Latest consensus state: current_term: 1 leader_uuid: "bb897562d0de40109d5696f1b3b85905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb897562d0de40109d5696f1b3b85905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260924 19:31:49.329802 4001 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [sys.catalog]: This master's current role is: LEADER
W20260924 19:31:49.329819 4015 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:31:49.329864 4015 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:31:49.329588 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb897562d0de40109d5696f1b3b85905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb897562d0de40109d5696f1b3b85905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34859 } } }
I20260924 19:31:49.329909 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:49.330060 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:49.330207 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:49.331682 4017 catalog_manager.cc:1347] Generated new cluster ID: 76ad2337ac6b4088b2d7789be0580676
I20260924 19:31:49.331732 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:49.337208 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.326872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52166 (local address 127.0.98.62:34859)
0924 19:31:49.327079 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:49.327088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:49.327148 (+ 60us) server_negotiation.cc:407] Connection header received
0924 19:31:49.327499 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:49.327505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:49.327551 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:49.327703 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:49.328509 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.330050 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.330841 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.331439 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.331549 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.333863 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:49.333868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:49.333868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:49.333880 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:49.335549 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.335888 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.335891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.335891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.335993 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.336184 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.336185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.336185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.336417 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:49.336584 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:49.336765 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 19:31:49.336908 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260924 19:31:49.339679 4017 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:49.340358 4017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:49.344789 4017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb897562d0de40109d5696f1b3b85905: Generated new TSK 0
I20260924 19:31:49.344940 4017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:49.372337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:49.453133 4021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:49.453329 4021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:49.453351 4021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:49.454435 4021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:49.454468 4021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:49.454493 4021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:49.455888 4021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:49.456065 4021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:49.456239 4021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:49.461179 4029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:49.461172 4026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:49.461174 4027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:49.461203 4021 server_base.cc:1034] running on GCE node
I20260924 19:31:49.461566 4021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:49.461786 4021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:49.462934 4021 hybrid_clock.cc:648] HybridClock initialized: now 1790278309462921 us; error 24 us; skew 500 ppm
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:49.465430 4021 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:49.465855 4021 webserver.cc:460] Webserver started at http://127.0.98.1:46171/ using document root <none> and password file <none>
I20260924 19:31:49.466051 4021 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:49.466099 4021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:49.466223 4021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:49.467275 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "082634767c084ca89b132879bb38d454"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "ddaddf0e4ee6e40f50b2fbba217baa8f"
server_key_iv: "059c8f295bd513d1dc67c7b3a188affb"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.467615 4021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "082634767c084ca89b132879bb38d454"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "ddaddf0e4ee6e40f50b2fbba217baa8f"
server_key_iv: "059c8f295bd513d1dc67c7b3a188affb"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.468956 4021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:31:49.469646 4036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.469811 4021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:49.469892 4021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "082634767c084ca89b132879bb38d454"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "ddaddf0e4ee6e40f50b2fbba217baa8f"
server_key_iv: "059c8f295bd513d1dc67c7b3a188affb"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.469952 4021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:49.481180 4021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:49.483085 4021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:49.483225 4021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:49.483445 4021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:49.483732 4021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:49.483764 4021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.483793 4021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:49.483808 4021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.489176 4021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33255
I20260924 19:31:49.489226 4149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33255 every 8 connection(s)
I20260924 19:31:49.489537 4021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:49.497108 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 4021
I20260924 19:31:49.497195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:49.497364 392 external_mini_cluster.cc:1428] Setting key f787f52464ccce257a98d1900b5180a5
I20260924 19:31:49.498767 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.490222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43181 (local address 127.0.98.62:34859)
0924 19:31:49.490278 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:49.490280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:49.490447 (+ 167us) server_negotiation.cc:407] Connection header received
0924 19:31:49.491385 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:49.491386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:49.491398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:49.491448 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:49.492046 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.492541 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.493543 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.493741 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.493797 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.495796 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:49.495798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:49.495798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:49.495806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:49.497329 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.497758 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.497759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.497759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.497820 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.498098 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.498099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.498099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.498238 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:49.498303 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:49.498593 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 19:31:49.498675 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:31:49.498867 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.490194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.1:43181)
0924 19:31:49.490332 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:49.490368 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:49.491278 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:49.491517 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:49.491520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:49.491775 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:49.491956 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:49.491960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.492623 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.492624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:49.493457 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:49.493459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.493813 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.493814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:49.493927 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.494350 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:49.494358 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:49.495674 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:49.497401 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:49.497407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:49.497410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:49.497693 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:49.497895 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:49.497896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:49.497896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:49.498027 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:49.498336 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:49.498342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:49.498462 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 19:31:49.498559 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:31:49.498696 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 19:31:49.499692 4150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260924 19:31:49.499775 4150 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:49.499935 4150 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260924 19:31:49.500488 3959 ts_manager.cc:194] Registered new tserver with Master: 082634767c084ca89b132879bb38d454 (127.0.98.1:33255)
I20260924 19:31:49.501392 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43181
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:49.523557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:49.602322 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:49.602527 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:49.602548 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:49.603633 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:49.603680 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:49.603708 4155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:49.605145 4155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:49.605327 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:49.605501 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:49.611110 4163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:49.615350 4161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:49.615518 4160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:49.615612 4155 server_base.cc:1034] running on GCE node
I20260924 19:31:49.615892 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:49.616194 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:49.617295 4155 hybrid_clock.cc:648] HybridClock initialized: now 1790278309617273 us; error 35 us; skew 500 ppm
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:49.620383 4155 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:49.620895 4155 webserver.cc:460] Webserver started at http://127.0.98.2:40167/ using document root <none> and password file <none>
I20260924 19:31:49.621100 4155 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:49.621147 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:49.621280 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:49.622290 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "13e825214631476e9039e965c76391ee"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "d2a81c7154c299b1773f7bc59accba8a"
server_key_iv: "f445c2b08574cf7f3f6be420d346d60e"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.622618 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "13e825214631476e9039e965c76391ee"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "d2a81c7154c299b1773f7bc59accba8a"
server_key_iv: "f445c2b08574cf7f3f6be420d346d60e"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.623807 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:49.624428 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.624588 4155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:49.624689 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "13e825214631476e9039e965c76391ee"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "d2a81c7154c299b1773f7bc59accba8a"
server_key_iv: "f445c2b08574cf7f3f6be420d346d60e"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.624759 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:49.633124 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:49.634752 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:49.634881 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:49.635068 4155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:49.635363 4155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:49.635396 4155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.635423 4155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:49.635439 4155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.640856 4155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40219
I20260924 19:31:49.640914 4283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40219 every 8 connection(s)
I20260924 19:31:49.641193 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:49.647991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 4155
I20260924 19:31:49.648087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:49.648250 392 external_mini_cluster.cc:1428] Setting key f882365b7ee8b39b5d1551efb0e690a0
I20260924 19:31:49.650183 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.641828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38127 (local address 127.0.98.62:34859)
0924 19:31:49.641905 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:49.641907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:49.642041 (+ 134us) server_negotiation.cc:407] Connection header received
0924 19:31:49.642951 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:49.642953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:49.642964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:49.643020 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:49.643583 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.644111 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.645148 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.645345 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.645403 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.647325 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:49.647327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:49.647327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:49.647336 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:49.648853 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.649305 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.649306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.649306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.649362 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.649613 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.649614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.649614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.649711 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:49.649757 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:49.650041 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 19:31:49.650109 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 19:31:49.650411 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.641789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.2:38127)
0924 19:31:49.641926 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:49.641962 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:49.642833 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:49.643072 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:49.643076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:49.643320 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:49.643502 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:49.643507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.644206 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.644207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:49.645073 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:49.645076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.645411 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.645412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:49.645502 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.645906 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:49.645914 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:49.647219 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:49.648930 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:49.648937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:49.648939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:49.649221 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:49.649426 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:49.649427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:49.649427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:49.649559 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:49.649762 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:49.649767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:49.649920 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:31:49.650027 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:31:49.650187 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260924 19:31:49.650868 4284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260924 19:31:49.650976 4284 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:49.651160 4284 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260924 19:31:49.651512 3959 ts_manager.cc:194] Registered new tserver with Master: 13e825214631476e9039e965c76391ee (127.0.98.2:40219)
I20260924 19:31:49.651898 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38127
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:49.672351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:49.751876 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:49.752077 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:49.752099 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:49.753293 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:49.753333 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:49.753361 4289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:49.754819 4289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:49.755029 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:49.755254 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:49.760123 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:49.760111 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:49.760193 4289 server_base.cc:1034] running on GCE node
W20260924 19:31:49.760111 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:49.760588 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:49.760902 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:49.762077 4289 hybrid_clock.cc:648] HybridClock initialized: now 1790278309762060 us; error 34 us; skew 500 ppm
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:49.765658 4289 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:49.766222 4289 webserver.cc:460] Webserver started at http://127.0.98.3:40215/ using document root <none> and password file <none>
I20260924 19:31:49.766484 4289 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:49.766547 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:49.766693 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:49.767781 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "334e4cc1b4c4419996cde768d65d26a6"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "85555319a1691f08cce699c84fb8866e"
server_key_iv: "082fec752e3e2b66abbdaf1c03a51c0f"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.768097 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "334e4cc1b4c4419996cde768d65d26a6"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "85555319a1691f08cce699c84fb8866e"
server_key_iv: "082fec752e3e2b66abbdaf1c03a51c0f"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.769475 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:31:49.770273 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.770452 4289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:49.770545 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "334e4cc1b4c4419996cde768d65d26a6"
format_stamp: "Formatted at 2026-09-24 19:31:49 on dist-test-slave-m8jb"
server_key: "85555319a1691f08cce699c84fb8866e"
server_key_iv: "082fec752e3e2b66abbdaf1c03a51c0f"
server_key_version: "encryptionkey@0"
I20260924 19:31:49.770612 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:49.788311 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:49.790365 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:49.790499 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:49.790709 4289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:49.791025 4289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:49.791064 4289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.791093 4289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:49.791117 4289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:49.797429 4289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41805
I20260924 19:31:49.797505 4417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41805 every 8 connection(s)
I20260924 19:31:49.797788 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:49.807128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 4289
I20260924 19:31:49.807245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:49.807447 392 external_mini_cluster.cc:1428] Setting key af7f79338b433522e6ccb3e26592ac44
I20260924 19:31:49.808602 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.798510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34953 (local address 127.0.98.62:34859)
0924 19:31:49.798582 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:49.798585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:49.798849 (+ 264us) server_negotiation.cc:407] Connection header received
0924 19:31:49.800063 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:49.800065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:49.800075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:49.800113 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:49.801039 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.801528 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.803014 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.803212 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.803266 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.805755 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:49.805756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:49.805756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:49.805763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:49.807068 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.807538 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.807539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.807539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.807591 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.807968 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.807969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.807970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.808106 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:49.808168 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:49.808480 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 19:31:49.808541 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:31:49.808894 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.798473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34859 (local address 127.0.98.3:34953)
0924 19:31:49.798661 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:49.798711 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:49.799952 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:49.800209 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:49.800214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:49.800606 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:49.800939 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:49.800946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.801617 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.801618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:49.802918 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:49.802922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.803300 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.803301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:49.803451 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.804001 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:49.804011 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:49.805662 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:49.807142 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:49.807149 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:49.807151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:49.807431 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:49.807670 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:49.807671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:49.807671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:49.807823 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:49.808170 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:49.808176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:49.808343 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 19:31:49.808464 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 19:31:49.808665 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260924 19:31:49.809422 4418 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34859
I20260924 19:31:49.809523 4418 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:49.809723 4418 heartbeater.cc:511] Master 127.0.98.62:34859 requested a full tablet report, sending...
I20260924 19:31:49.810082 3959 ts_manager.cc:194] Registered new tserver with Master: 334e4cc1b4c4419996cde768d65d26a6 (127.0.98.3:41805)
I20260924 19:31:49.810572 3959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34953
I20260924 19:31:49.819177 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:49.845942 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.838475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.0.98.1:33255)
0924 19:31:49.838659 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:49.838664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:49.838676 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:31:49.838704 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:49.838705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:49.838721 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:49.838792 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:49.839281 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.839996 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.840729 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.841070 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.841125 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.843228 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:49.843236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:49.843236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:49.843243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:49.844720 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.845028 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.845031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.845031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.845085 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.845293 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.845294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.845295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.845487 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:49.845636 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:49.845756 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:31:49.845835 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[3914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:49.853204 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:49.847113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.0.98.62:34859)
0924 19:31:49.847186 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:49.847188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:49.847198 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:31:49.847220 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:49.847220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:49.847229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:49.847275 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:49.847664 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.848123 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.848834 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:49.849012 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:49.849058 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:49.850961 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:49.850963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:49.850963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:49.850970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:49.852133 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.852423 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.852424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.852425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.852475 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:49.852741 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:49.852742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:49.852742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:49.852895 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:49.852974 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:49.853077 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:31:49.853134 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 19:31:49.853473 3959 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52180
I20260924 19:31:49.853827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 4021
I20260924 19:31:49.857789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 4155
I20260924 19:31:49.861615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 4289
I20260924 19:31:49.865226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 3927
2026-09-24T19:31:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2835 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[4430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[4430](info): set up 2 sockets
Sep 24 19:31:49 dist-test-slave-m8jb krb5kdc[4430](info): commencing operation
krb5kdc: starting...
W20260924 19:31:50.824363 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:50 dist-test-slave-m8jb krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:50.935709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:51.013906 4443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:51.014114 4443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:51.014145 4443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:51.015290 4443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:51.015329 4443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:51.015347 4443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:51.015363 4443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:51.015378 4443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:51.016889 4443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:51.017146 4443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:51.017356 4443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:51.021867 4448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.021917 4449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.022068 4451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:51.022073 4443 server_base.cc:1034] running on GCE node
I20260924 19:31:51.022359 4443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:51.022641 4443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:51.023787 4443 hybrid_clock.cc:648] HybridClock initialized: now 1790278311023773 us; error 34 us; skew 500 ppm
Sep 24 19:31:51 dist-test-slave-m8jb krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:51.026271 4443 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:51.026612 4443 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:51.026665 4443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:51.026763 4443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:51.027813 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "ae6ffff9841a44d5a2c6a23600999ae7"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "d2b78f558199912b2a5f8284e2a15119"
server_key_iv: "349173c7d614d82ecbc9c6483dedce26"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.028131 4443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae6ffff9841a44d5a2c6a23600999ae7"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "d2b78f558199912b2a5f8284e2a15119"
server_key_iv: "349173c7d614d82ecbc9c6483dedce26"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.029448 4443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:51.030236 4457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.030426 4443 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:51.030499 4443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "ae6ffff9841a44d5a2c6a23600999ae7"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "d2b78f558199912b2a5f8284e2a15119"
server_key_iv: "349173c7d614d82ecbc9c6483dedce26"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.030551 4443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:51.052881 4443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:51.055286 4443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:51.055408 4443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:51.058965 4443 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35185
I20260924 19:31:51.059032 4509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35185 every 8 connection(s)
I20260924 19:31:51.059270 4443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:51.060012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 4443
I20260924 19:31:51.060098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:51.060230 4510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:51.060252 392 external_mini_cluster.cc:1428] Setting key f89da57fabb3bb010075a8aec88b7b33
I20260924 19:31:51.063532 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7: Bootstrap starting.
I20260924 19:31:51.064281 4510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:51.064555 4510 log.cc:824] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:51.065342 4510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7: No bootstrap required, opened a new log
Sep 24 19:31:51 dist-test-slave-m8jb krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:51.067751 4510 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ffff9841a44d5a2c6a23600999ae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35185 } }
I20260924 19:31:51.067921 4510 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:51.067965 4510 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6ffff9841a44d5a2c6a23600999ae7, State: Initialized, Role: FOLLOWER
I20260924 19:31:51.068084 4510 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ffff9841a44d5a2c6a23600999ae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35185 } }
I20260924 19:31:51.068164 4510 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:51.068202 4510 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:51.068241 4510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:51.068884 4510 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ffff9841a44d5a2c6a23600999ae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35185 } }
I20260924 19:31:51.068992 4510 leader_election.cc:302] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6ffff9841a44d5a2c6a23600999ae7; no voters:
I20260924 19:31:51.069164 4510 leader_election.cc:288] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:51.069321 4516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:51.069416 4510 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:51.069518 4516 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [term 1 LEADER]: Becoming Leader. State: Replica: ae6ffff9841a44d5a2c6a23600999ae7, State: Running, Role: LEADER
I20260924 19:31:51.069643 4516 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ffff9841a44d5a2c6a23600999ae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35185 } }
I20260924 19:31:51.069993 4516 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6ffff9841a44d5a2c6a23600999ae7. Latest consensus state: current_term: 1 leader_uuid: "ae6ffff9841a44d5a2c6a23600999ae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ffff9841a44d5a2c6a23600999ae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35185 } } }
I20260924 19:31:51.070133 4516 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:51.070250 4515 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6ffff9841a44d5a2c6a23600999ae7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ffff9841a44d5a2c6a23600999ae7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35185 } } }
I20260924 19:31:51.070338 4515 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:51.070484 4524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:51.070981 4524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:51.071801 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.060875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.0.98.62:35185)
0924 19:31:51.061052 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.061061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.061084 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:31:51.061492 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.061499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.061544 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.061652 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:51.062449 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.063333 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.064356 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.064849 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.064958 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.067599 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:51.067604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:51.067604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:51.067617 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:51.069843 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.070229 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.070232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.070232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.070337 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.070549 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.070550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.070550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.070853 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:51.070985 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.071462 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.071598 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260924 19:31:51.072996 4524 catalog_manager.cc:1347] Generated new cluster ID: dda1d78f1b1446fbbc9e9cb2ecdf8e16
I20260924 19:31:51.073052 4524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:51.110520 4524 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:51.111300 4524 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:51.121181 4524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7: Generated new TSK 0
I20260924 19:31:51.121393 4524 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:51.148365 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35185
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:51.227314 4536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:51.227514 4536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:51.227535 4536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:51.228708 4536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:51.228746 4536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:51.228772 4536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:51.230265 4536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:51.230484 4536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:51.230686 4536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:51.235558 4541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:51.235633 4536 server_base.cc:1034] running on GCE node
W20260924 19:31:51.235538 4544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.235541 4542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:51.235958 4536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:51.236148 4536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:51.237300 4536 hybrid_clock.cc:648] HybridClock initialized: now 1790278311237287 us; error 31 us; skew 500 ppm
Sep 24 19:31:51 dist-test-slave-m8jb krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:51.239552 4536 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:51.239861 4536 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:51.239905 4536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:51.240012 4536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:51.241029 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "8dc05bdf06eb4439be72b5fe073790ae"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "9e60916e8008c5f73cc845d26594effd"
server_key_iv: "c314e0e9dcd7295d135e6c07d0b18db5"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.241348 4536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8dc05bdf06eb4439be72b5fe073790ae"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "9e60916e8008c5f73cc845d26594effd"
server_key_iv: "c314e0e9dcd7295d135e6c07d0b18db5"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.242489 4536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:51.243180 4550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.243343 4536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:51.243423 4536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "8dc05bdf06eb4439be72b5fe073790ae"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "9e60916e8008c5f73cc845d26594effd"
server_key_iv: "c314e0e9dcd7295d135e6c07d0b18db5"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.243481 4536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:51.254873 4536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:51.257025 4536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:51.257160 4536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:51.257372 4536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:51.257663 4536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:51.257699 4536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.257728 4536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:51.257745 4536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.262944 4536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39247
I20260924 19:31:51.263017 4663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39247 every 8 connection(s)
I20260924 19:31:51.263262 4536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:31:51 dist-test-slave-m8jb krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:51.271994 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.263986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46829 (local address 127.0.98.62:35185)
0924 19:31:51.264043 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.264045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.264227 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:31:51.264742 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.264743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.264754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.264807 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:51.265728 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.266227 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.267217 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.267430 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.267484 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.269488 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:51.269489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:51.269490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:51.269497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:51.270825 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.271188 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.271188 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.271189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.271243 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.271435 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.271436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.271436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.271537 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:51.271583 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.271833 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.271919 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:31:51.272082 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.263945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35185 (local address 127.0.98.1:46829)
0924 19:31:51.264107 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.264146 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.264621 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.264908 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.264911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.265153 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:51.265627 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.265632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.266319 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.266320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.267128 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.267130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.267502 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.267503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.267621 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.268057 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:51.268065 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:51.269391 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:51.270885 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:51.270891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:51.270893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:51.271119 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:51.271298 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:51.271298 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:51.271298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:51.271392 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:51.271579 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:51.271581 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:51.271693 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.271770 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.271909 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260924 19:31:51.272619 4664 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35185
I20260924 19:31:51.272738 4664 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:51.272857 4664 heartbeater.cc:511] Master 127.0.98.62:35185 requested a full tablet report, sending...
I20260924 19:31:51.273319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 4536
I20260924 19:31:51.273396 4474 ts_manager.cc:194] Registered new tserver with Master: 8dc05bdf06eb4439be72b5fe073790ae (127.0.98.1:39247)
I20260924 19:31:51.273430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:51.273618 392 external_mini_cluster.cc:1428] Setting key b44abb44aa22efdd16e26ff84fbec5d7
I20260924 19:31:51.274233 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46829
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:51.299073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35185
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:51.377696 4669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:51.377907 4669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:51.377930 4669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:51.379112 4669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:51.379153 4669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:51.379180 4669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:51.380712 4669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:51.380923 4669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:51.381140 4669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:51.385993 4675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.385951 4674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.385946 4677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:51.386063 4669 server_base.cc:1034] running on GCE node
I20260924 19:31:51.386343 4669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:51.386561 4669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:51.387708 4669 hybrid_clock.cc:648] HybridClock initialized: now 1790278311387694 us; error 26 us; skew 500 ppm
Sep 24 19:31:51 dist-test-slave-m8jb krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:51.390131 4669 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:51.390440 4669 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:51.390486 4669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:51.390578 4669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:51.391556 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "4722eeb53d954a37bec42a03f00ead9f"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "6aa0f1609fa2b71fa74a78320d6d5c23"
server_key_iv: "fc5abf81955881e5afe8a859a716b642"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.391839 4669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4722eeb53d954a37bec42a03f00ead9f"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "6aa0f1609fa2b71fa74a78320d6d5c23"
server_key_iv: "fc5abf81955881e5afe8a859a716b642"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.392995 4669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:51.393656 4683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.393823 4669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:51.393903 4669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "4722eeb53d954a37bec42a03f00ead9f"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "6aa0f1609fa2b71fa74a78320d6d5c23"
server_key_iv: "fc5abf81955881e5afe8a859a716b642"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.393957 4669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:51.408495 4669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:51.410712 4669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:51.410863 4669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:51.411087 4669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:51.411382 4669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:51.411414 4669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.411444 4669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:51.411463 4669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.416724 4669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34841
I20260924 19:31:51.416786 4796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34841 every 8 connection(s)
I20260924 19:31:51.417030 4669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:31:51 dist-test-slave-m8jb krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:51.423578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 4669
I20260924 19:31:51.423684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:51.423882 392 external_mini_cluster.cc:1428] Setting key 408adb4ab5889d358d60521827477609
I20260924 19:31:51.426000 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.417699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47733 (local address 127.0.98.62:35185)
0924 19:31:51.417762 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.417764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.417960 (+ 196us) server_negotiation.cc:407] Connection header received
0924 19:31:51.418463 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.418464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.418477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.418524 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:51.419457 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.419952 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.420990 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.421197 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.421254 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.423267 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:51.423268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:51.423268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:51.423276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:51.424733 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.425093 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.425094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.425094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.425156 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.425359 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.425359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.425360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.425485 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:51.425549 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.425854 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.425937 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:31:51.426215 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.417664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35185 (local address 127.0.98.2:47733)
0924 19:31:51.417847 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.417883 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.418365 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.418604 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.418608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.418873 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:51.419358 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.419363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.420044 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.420045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.420912 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.420915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.421269 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.421270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.421391 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.421873 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:51.421881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:51.423179 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:51.424800 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:51.424806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:51.424808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:51.425022 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:51.425213 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:51.425214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:51.425214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:51.425308 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:51.425539 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:51.425541 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:51.425696 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.425819 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.425989 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260924 19:31:51.426661 4797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35185
I20260924 19:31:51.426756 4797 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:51.426903 4797 heartbeater.cc:511] Master 127.0.98.62:35185 requested a full tablet report, sending...
I20260924 19:31:51.427284 4474 ts_manager.cc:194] Registered new tserver with Master: 4722eeb53d954a37bec42a03f00ead9f (127.0.98.2:34841)
I20260924 19:31:51.427647 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47733
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:51.448004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35185
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:51.525481 4802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:51.525692 4802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:51.525715 4802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:51.526859 4802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:51.526899 4802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:51.526924 4802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:51.528398 4802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:51.528650 4802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:51.528860 4802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:51.533540 4810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.533558 4808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.533540 4807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:51.533878 4802 server_base.cc:1034] running on GCE node
I20260924 19:31:51.534077 4802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:51.534327 4802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:51.535470 4802 hybrid_clock.cc:648] HybridClock initialized: now 1790278311535456 us; error 30 us; skew 500 ppm
Sep 24 19:31:51 dist-test-slave-m8jb krb5kdc[4430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:51.541771 4802 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:51.542129 4802 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:51.542182 4802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:51.542300 4802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:51.543314 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "a25840050655433b871b29911fd46ce7"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "a17a0e3769b6188182d0028162b4a853"
server_key_iv: "e93b4a5eb9f1b90eff03a21ac170877c"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.543635 4802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a25840050655433b871b29911fd46ce7"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "a17a0e3769b6188182d0028162b4a853"
server_key_iv: "e93b4a5eb9f1b90eff03a21ac170877c"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.544854 4802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:51.545485 4816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.545667 4802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:51.545747 4802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "a25840050655433b871b29911fd46ce7"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "a17a0e3769b6188182d0028162b4a853"
server_key_iv: "e93b4a5eb9f1b90eff03a21ac170877c"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.545807 4802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:51.570271 4802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:51.572537 4802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:51.572745 4802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:51.572953 4802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:51.573241 4802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:51.573273 4802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.573302 4802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:51.573319 4802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.578717 4802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34641
I20260924 19:31:51.578751 4929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34641 every 8 connection(s)
I20260924 19:31:51.578994 4802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:31:51.582392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 4802
I20260924 19:31:51.582489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:51.582789 392 external_mini_cluster.cc:1428] Setting key 8b50241d439c32aba8fa28ab489e8279
Sep 24 19:31:51 dist-test-slave-m8jb krb5kdc[4430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:51.587747 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.579647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42471 (local address 127.0.98.62:35185)
0924 19:31:51.579728 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.579730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.579883 (+ 153us) server_negotiation.cc:407] Connection header received
0924 19:31:51.580352 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.580353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.580363 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.580415 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:51.581446 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.581956 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.583247 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.583446 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.583503 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.585466 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:51.585467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:51.585468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:51.585475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:51.586522 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.586889 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.586891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.586891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.586944 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.587154 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.587155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.587155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.587266 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:51.587321 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.587582 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.587659 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 19:31:51.587930 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.579620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35185 (local address 127.0.98.3:42471)
0924 19:31:51.579753 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.579790 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.580266 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.580473 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.580476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.580768 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:51.581266 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.581272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.582030 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.582030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.583163 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.583166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.583518 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.583519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.583632 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.584055 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:51.584063 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:51.585376 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:51.586599 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:51.586605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:51.586608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:51.586813 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:51.586999 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:51.586999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:51.587000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:51.587097 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:51.587322 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:51.587328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:51.587465 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.587552 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.587704 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260924 19:31:51.588362 4930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35185
I20260924 19:31:51.588449 4930 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:51.588563 4930 heartbeater.cc:511] Master 127.0.98.62:35185 requested a full tablet report, sending...
I20260924 19:31:51.588903 4474 ts_manager.cc:194] Registered new tserver with Master: a25840050655433b871b29911fd46ce7 (127.0.98.3:34641)
I20260924 19:31:51.589233 4474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42471
I20260924 19:31:51.594401 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:31:51.601191 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.595735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.0.98.62:35185)
0924 19:31:51.595810 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.595812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.595874 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:31:51.595940 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.595941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.595950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.595995 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:51.596543 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.597094 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.597823 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.598018 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.598069 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.599149 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:51.599150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:51.599151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:51.599158 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:51.600128 (+ 970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.600443 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.600443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.600444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.600499 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:51.600677 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:51.600678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:51.600678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:51.600823 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:51.600912 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.601028 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.601095 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:31:51.602453 4474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:51.603988 4474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:51.610592 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.605938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34841 (local address 127.0.0.1:44404)
0924 19:31:51.606097 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.606136 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.606197 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.606627 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.606627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.606629 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:51.606715 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.606717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.607984 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.607985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.609381 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.609382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.610289 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.610290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.610374 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.610391 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.610437 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.610503 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":106,"thread_start_us":72,"threads_started":1}
I20260924 19:31:51.610594 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.605938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39247 (local address 127.0.0.1:40074)
0924 19:31:51.606291 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.606299 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.606333 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.606640 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.606641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.606644 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:51.606690 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.606692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.608058 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.608059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.609389 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.609392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.610341 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.610342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.610408 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.610413 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.610442 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.610536 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":71,"thread_start_us":65,"threads_started":1}
I20260924 19:31:51.610822 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.605941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34641 (local address 127.0.0.1:41454)
0924 19:31:51.606170 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.606182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.606229 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.606530 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.606533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.606535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:51.606601 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.606603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.607928 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.607928 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.609372 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.609375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.610291 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.610292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.610374 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.610391 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.610437 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.610505 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":112,"thread_start_us":48,"threads_started":1}
I20260924 19:31:51.610982 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.606194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.0.98.3:34641)
0924 19:31:51.606354 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.606360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.606370 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:31:51.606409 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.606411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.606426 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.606509 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:51.606678 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.607809 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.609466 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.610196 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.610614 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.610837 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.610863 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.610904 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260924 19:31:51.611088 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.606309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44404 (local address 127.0.98.2:34841)
0924 19:31:51.606464 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.606468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.606476 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:51.606515 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.606517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.606532 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.606786 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:51.606861 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.607890 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.609466 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.610190 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.610901 (+ 711us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.610958 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.610981 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.611017 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260924 19:31:51.611352 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.606119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.98.1:39247)
0924 19:31:51.606511 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.606515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.606524 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:51.606558 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.606559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.606573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.606640 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:51.606822 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.607910 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.609494 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.610228 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.610485 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.610555 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.610580 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.610620 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":76,"threads_started":1}
I20260924 19:31:51.611864 4731 tablet_service.cc:1467] Processing CreateTablet for tablet 745c947135744077bd80bba6d4f629e6 (DEFAULT_TABLE table=test-table [id=7cf787a8927546fcb936e7b2a2f5d0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:51.611920 4598 tablet_service.cc:1467] Processing CreateTablet for tablet 745c947135744077bd80bba6d4f629e6 (DEFAULT_TABLE table=test-table [id=7cf787a8927546fcb936e7b2a2f5d0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:51.612215 4731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745c947135744077bd80bba6d4f629e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:51.612215 4598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745c947135744077bd80bba6d4f629e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:51.613387 4864 tablet_service.cc:1467] Processing CreateTablet for tablet 745c947135744077bd80bba6d4f629e6 (DEFAULT_TABLE table=test-table [id=7cf787a8927546fcb936e7b2a2f5d0c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:51.613817 4864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745c947135744077bd80bba6d4f629e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:51.616289 4945 tablet_bootstrap.cc:492] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: Bootstrap starting.
I20260924 19:31:51.616421 4946 tablet_bootstrap.cc:492] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: Bootstrap starting.
I20260924 19:31:51.616616 4944 tablet_bootstrap.cc:492] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae: Bootstrap starting.
I20260924 19:31:51.616997 4946 tablet_bootstrap.cc:654] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:51.617007 4945 tablet_bootstrap.cc:654] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:51.617223 4944 tablet_bootstrap.cc:654] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:51.617494 4946 log.cc:824] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:51.617570 4945 log.cc:824] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:51.617761 4944 log.cc:824] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:51.618163 4946 tablet_bootstrap.cc:492] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: No bootstrap required, opened a new log
I20260924 19:31:51.618221 4946 ts_tablet_manager.cc:1397] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:31:51.618258 4945 tablet_bootstrap.cc:492] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: No bootstrap required, opened a new log
I20260924 19:31:51.618325 4945 ts_tablet_manager.cc:1397] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:31:51.618374 4944 tablet_bootstrap.cc:492] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae: No bootstrap required, opened a new log
I20260924 19:31:51.618430 4944 ts_tablet_manager.cc:1397] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:31:51.619665 4946 raft_consensus.cc:348] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.619796 4946 raft_consensus.cc:374] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:51.619836 4946 raft_consensus.cc:729] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4722eeb53d954a37bec42a03f00ead9f, State: Initialized, Role: FOLLOWER
I20260924 19:31:51.619951 4946 consensus_queue.cc:260] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.620193 4946 ts_tablet_manager.cc:1428] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:31:51.620298 4797 heartbeater.cc:503] Master 127.0.98.62:35185 was elected leader, sending a full tablet report...
I20260924 19:31:51.620217 4945 raft_consensus.cc:348] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.620275 4944 raft_consensus.cc:348] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.620379 4945 raft_consensus.cc:374] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:51.620398 4944 raft_consensus.cc:374] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:51.620424 4945 raft_consensus.cc:729] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25840050655433b871b29911fd46ce7, State: Initialized, Role: FOLLOWER
I20260924 19:31:51.620428 4944 raft_consensus.cc:729] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc05bdf06eb4439be72b5fe073790ae, State: Initialized, Role: FOLLOWER
I20260924 19:31:51.620584 4944 consensus_queue.cc:260] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.620584 4945 consensus_queue.cc:260] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.620908 4944 ts_tablet_manager.cc:1428] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 19:31:51.620906 4945 ts_tablet_manager.cc:1428] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 19:31:51.621057 4664 heartbeater.cc:503] Master 127.0.98.62:35185 was elected leader, sending a full tablet report...
I20260924 19:31:51.621272 4930 heartbeater.cc:503] Master 127.0.98.62:35185 was elected leader, sending a full tablet report...
I20260924 19:31:51.635167 4952 raft_consensus.cc:482] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:51.635281 4952 raft_consensus.cc:504] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.635661 4952 leader_election.cc:288] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4722eeb53d954a37bec42a03f00ead9f (127.0.98.2:34841), a25840050655433b871b29911fd46ce7 (127.0.98.3:34641)
I20260924 19:31:51.638494 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.635911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58889 (local address 127.0.98.2:34841)
0924 19:31:51.635953 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.635956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.635980 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:31:51.636007 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.636008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.636021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.636068 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:51.636250 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.636805 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.637785 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.638155 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.638205 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.638214 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.638388 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.638427 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:31:51.638489 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.635798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34841 (local address 127.0.98.1:58889)
0924 19:31:51.635855 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.635869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.635925 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.636104 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.636105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.636107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:51.636187 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.636189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.636881 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.636883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.637699 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.637702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.638238 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.638239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.638311 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.638314 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.638372 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.638420 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:31:51.638562 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.635914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37365 (local address 127.0.98.3:34641)
0924 19:31:51.635961 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.635962 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.636085 (+ 123us) server_negotiation.cc:407] Connection header received
0924 19:31:51.636135 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.636136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.636148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.636213 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:51.636350 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.636912 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.637855 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.638241 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.638307 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.638317 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.638467 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.638504 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:31:51.638612 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.635798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34641 (local address 127.0.98.1:37365)
0924 19:31:51.635996 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.636005 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.636061 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.636219 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.636219 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.636221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:51.636293 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.636295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.637034 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.637035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.637770 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:51.637772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.638315 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.638316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:51.638397 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.638401 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.638453 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.638490 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":13,"thread_start_us":127,"threads_started":1}
I20260924 19:31:51.638816 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c947135744077bd80bba6d4f629e6" candidate_uuid: "8dc05bdf06eb4439be72b5fe073790ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4722eeb53d954a37bec42a03f00ead9f" is_pre_election: true
I20260924 19:31:51.638896 4884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c947135744077bd80bba6d4f629e6" candidate_uuid: "8dc05bdf06eb4439be72b5fe073790ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25840050655433b871b29911fd46ce7" is_pre_election: true
I20260924 19:31:51.638988 4751 raft_consensus.cc:2454] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc05bdf06eb4439be72b5fe073790ae in term 0.
I20260924 19:31:51.639019 4884 raft_consensus.cc:2454] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc05bdf06eb4439be72b5fe073790ae in term 0.
I20260924 19:31:51.639176 4552 leader_election.cc:302] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4722eeb53d954a37bec42a03f00ead9f, 8dc05bdf06eb4439be72b5fe073790ae; no voters:
I20260924 19:31:51.639302 4952 raft_consensus.cc:2781] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:51.639367 4952 raft_consensus.cc:482] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:51.639384 4952 raft_consensus.cc:3037] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:51.640000 4952 raft_consensus.cc:504] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.640128 4952 leader_election.cc:288] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [CANDIDATE]: Term 1 election: Requested vote from peers 4722eeb53d954a37bec42a03f00ead9f (127.0.98.2:34841), a25840050655433b871b29911fd46ce7 (127.0.98.3:34641)
I20260924 19:31:51.640251 4751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c947135744077bd80bba6d4f629e6" candidate_uuid: "8dc05bdf06eb4439be72b5fe073790ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4722eeb53d954a37bec42a03f00ead9f"
I20260924 19:31:51.640269 4884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745c947135744077bd80bba6d4f629e6" candidate_uuid: "8dc05bdf06eb4439be72b5fe073790ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25840050655433b871b29911fd46ce7"
I20260924 19:31:51.640321 4751 raft_consensus.cc:3037] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:51.640327 4884 raft_consensus.cc:3037] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:51.641167 4751 raft_consensus.cc:2454] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc05bdf06eb4439be72b5fe073790ae in term 1.
I20260924 19:31:51.641163 4884 raft_consensus.cc:2454] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc05bdf06eb4439be72b5fe073790ae in term 1.
I20260924 19:31:51.641330 4553 leader_election.cc:302] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc05bdf06eb4439be72b5fe073790ae, a25840050655433b871b29911fd46ce7; no voters:
I20260924 19:31:51.641451 4952 raft_consensus.cc:2781] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:51.641588 4952 raft_consensus.cc:686] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 1 LEADER]: Becoming Leader. State: Replica: 8dc05bdf06eb4439be72b5fe073790ae, State: Running, Role: LEADER
I20260924 19:31:51.641671 4952 consensus_queue.cc:237] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } }
I20260924 19:31:51.642421 4474 catalog_manager.cc:5495] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dc05bdf06eb4439be72b5fe073790ae (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8dc05bdf06eb4439be72b5fe073790ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dc05bdf06eb4439be72b5fe073790ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 } health_report { overall_health: UNKNOWN } } }
I20260924 19:31:51.651687 4941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.649517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.0.98.1:39247)
0924 19:31:51.649580 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.649583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.649591 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:51.649621 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.649622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.649640 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.649708 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:51.649833 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.650296 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.651021 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:51.651204 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:51.651257 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:51.651398 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:51.651491 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:51.651558 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.651581 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.651600 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:31:51.653316 4751 raft_consensus.cc:1261] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 1 FOLLOWER]: Refusing update from remote peer 8dc05bdf06eb4439be72b5fe073790ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:51.653316 4884 raft_consensus.cc:1261] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 1 FOLLOWER]: Refusing update from remote peer 8dc05bdf06eb4439be72b5fe073790ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:51.653630 4954 consensus_queue.cc:1035] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25840050655433b871b29911fd46ce7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:51.653767 4952 consensus_queue.cc:1035] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4722eeb53d954a37bec42a03f00ead9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:51.655876 4957 mvcc.cc:205] Tried to move back new op lower bound from 7332979964529369088 to 7332979964484820992. Current Snapshot: MvccSnapshot[applied={T|T < 7332979964529369088}]
I20260924 19:31:51.656605 4958 mvcc.cc:205] Tried to move back new op lower bound from 7332979964529369088 to 7332979964484820992. Current Snapshot: MvccSnapshot[applied={T|T < 7332979964529369088}]
I20260924 19:31:51.656669 4959 mvcc.cc:205] Tried to move back new op lower bound from 7332979964529369088 to 7332979964484820992. Current Snapshot: MvccSnapshot[applied={T|T < 7332979964529369088}]
I20260924 19:31:51.658211 4473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:51.658309 4473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:51.659017 4473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6ffff9841a44d5a2c6a23600999ae7: Sending DeleteTablet for 3 replicas of tablet 745c947135744077bd80bba6d4f629e6
I20260924 19:31:51.659291 4598 tablet_service.cc:1514] Processing DeleteTablet for tablet 745c947135744077bd80bba6d4f629e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40074
I20260924 19:31:51.659302 4731 tablet_service.cc:1514] Processing DeleteTablet for tablet 745c947135744077bd80bba6d4f629e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44404
I20260924 19:31:51.659493 4966 tablet_replica.cc:330] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae: stopping tablet replica
I20260924 19:31:51.659508 4864 tablet_service.cc:1514] Processing DeleteTablet for tablet 745c947135744077bd80bba6d4f629e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41454
I20260924 19:31:51.659592 4966 raft_consensus.cc:2229] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:31:51.659686 4966 raft_consensus.cc:2258] T 745c947135744077bd80bba6d4f629e6 P 8dc05bdf06eb4439be72b5fe073790ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:51.659680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 4536
I20260924 19:31:51.659694 4967 tablet_replica.cc:330] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: stopping tablet replica
I20260924 19:31:51.659768 4967 raft_consensus.cc:2229] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:51.660199 4967 raft_consensus.cc:2258] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:51.660351 4968 tablet_replica.cc:330] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: stopping tablet replica
I20260924 19:31:51.660449 4968 raft_consensus.cc:2229] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:51.660538 4968 raft_consensus.cc:2258] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:51.660936 4968 ts_tablet_manager.cc:1905] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:51.661625 4967 ts_tablet_manager.cc:1905] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:51.662627 4968 ts_tablet_manager.cc:1918] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:51.662707 4968 log.cc:1196] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/745c947135744077bd80bba6d4f629e6
I20260924 19:31:51.662955 4968 ts_tablet_manager.cc:1939] T 745c947135744077bd80bba6d4f629e6 P a25840050655433b871b29911fd46ce7: Deleting consensus metadata
I20260924 19:31:51.663061 4967 ts_tablet_manager.cc:1918] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:51.663126 4967 log.cc:1196] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/745c947135744077bd80bba6d4f629e6
I20260924 19:31:51.663357 4967 ts_tablet_manager.cc:1939] T 745c947135744077bd80bba6d4f629e6 P 4722eeb53d954a37bec42a03f00ead9f: Deleting consensus metadata
I20260924 19:31:51.663367 4460 catalog_manager.cc:4841] TS a25840050655433b871b29911fd46ce7 (127.0.98.3:34641): tablet 745c947135744077bd80bba6d4f629e6 (table test-table [id=7cf787a8927546fcb936e7b2a2f5d0c1]) successfully deleted
I20260924 19:31:51.663770 4459 catalog_manager.cc:4841] TS 4722eeb53d954a37bec42a03f00ead9f (127.0.98.2:34841): tablet 745c947135744077bd80bba6d4f629e6 (table test-table [id=7cf787a8927546fcb936e7b2a2f5d0c1]) successfully deleted
W20260924 19:31:51.665093 4458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39247 (error 108)
I20260924 19:31:51.665299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 4669
I20260924 19:31:51.665887 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.665390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39247 (local address 127.0.0.1:34544)
0924 19:31:51.665806 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.665854 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
W20260924 19:31:51.666128 4458 catalog_manager.cc:4584] TS 8dc05bdf06eb4439be72b5fe073790ae (127.0.98.1:39247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 745c947135744077bd80bba6d4f629e6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39247: connect: Connection refused (error 111)
I20260924 19:31:51.669675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 4802
I20260924 19:31:51.673523 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 4443
2026-09-24T19:31:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1808 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T19:31:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:51Z Disabled control of system clock
I20260924 19:31:51.698513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37929
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:31:51.775668 4973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:51.775870 4973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:51.775890 4973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:51.777045 4973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:51.777086 4973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:51.777097 4973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:51.777108 4973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:51.777119 4973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:51.778435 4973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:51.778628 4973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:51.778859 4973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:51.783555 4979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.783674 4981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.783591 4978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:51.783582 4973 server_base.cc:1034] running on GCE node
I20260924 19:31:51.784010 4973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:51.784293 4973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:51.785427 4973 hybrid_clock.cc:648] HybridClock initialized: now 1790278311785416 us; error 26 us; skew 500 ppm
I20260924 19:31:51.786541 4973 webserver.cc:460] Webserver started at http://127.0.98.62:37619/ using document root <none> and password file <none>
I20260924 19:31:51.786756 4973 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:51.786803 4973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:51.786906 4973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:51.787976 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "353334607dd34c3584aa00f20367a91f"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "b816e59eaff1d1322cd6d26b540458c4"
server_key_iv: "b61876f6ef43cea3df13b92ca1aab29c"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.788304 4973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "353334607dd34c3584aa00f20367a91f"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "b816e59eaff1d1322cd6d26b540458c4"
server_key_iv: "b61876f6ef43cea3df13b92ca1aab29c"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.789649 4973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:31:51.790432 4987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.790565 4973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:51.790648 4973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "353334607dd34c3584aa00f20367a91f"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "b816e59eaff1d1322cd6d26b540458c4"
server_key_iv: "b61876f6ef43cea3df13b92ca1aab29c"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.790722 4973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:51.794106 4973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:51.794363 4973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:51.794486 4973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:51.798381 4973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37929
I20260924 19:31:51.798409 5039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37929 every 8 connection(s)
I20260924 19:31:51.798714 4973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:51.799310 5040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:51.802027 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f: Bootstrap starting.
I20260924 19:31:51.802178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 4973
I20260924 19:31:51.802266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:51.802472 392 external_mini_cluster.cc:1428] Setting key 923ccfb485dbfb1806fcf8417e2e72ee
I20260924 19:31:51.802719 5040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:51.803046 5040 log.cc:824] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:51.803758 5040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f: No bootstrap required, opened a new log
I20260924 19:31:51.805030 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.802999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.0.98.62:37929)
0924 19:31:51.803198 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.803205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.803229 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:31:51.803709 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.803717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.803764 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.803891 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:51.804545 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:51.804548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:51.804548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:51.804552 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:51.804593 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:51.804604 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:51.804743 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.804792 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.804845 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260924 19:31:51.805452 5040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353334607dd34c3584aa00f20367a91f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37929 } }
I20260924 19:31:51.805583 5040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:51.805614 5040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353334607dd34c3584aa00f20367a91f, State: Initialized, Role: FOLLOWER
I20260924 19:31:51.805733 5040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353334607dd34c3584aa00f20367a91f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37929 } }
I20260924 19:31:51.805809 5040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:51.805841 5040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:51.805888 5040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:51.806663 5040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353334607dd34c3584aa00f20367a91f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37929 } }
I20260924 19:31:51.806746 5040 leader_election.cc:302] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353334607dd34c3584aa00f20367a91f; no voters:
I20260924 19:31:51.806895 5040 leader_election.cc:288] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:51.806946 5045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:51.807089 5045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [term 1 LEADER]: Becoming Leader. State: Replica: 353334607dd34c3584aa00f20367a91f, State: Running, Role: LEADER
I20260924 19:31:51.807129 5040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:51.807189 5045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353334607dd34c3584aa00f20367a91f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37929 } }
I20260924 19:31:51.807507 5047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353334607dd34c3584aa00f20367a91f. Latest consensus state: current_term: 1 leader_uuid: "353334607dd34c3584aa00f20367a91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353334607dd34c3584aa00f20367a91f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37929 } } }
I20260924 19:31:51.807598 5047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:51.807502 5046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353334607dd34c3584aa00f20367a91f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353334607dd34c3584aa00f20367a91f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37929 } } }
I20260924 19:31:51.807739 5046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:51.807849 5053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:51.808298 5053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:51.809713 5053 catalog_manager.cc:1347] Generated new cluster ID: 93b9cde0ab5a4004984557f12aa4d4f2
I20260924 19:31:51.809755 5053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:51.817668 5053 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:51.817799 5053 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:51.824100 5053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f: Generated new TSK 0
I20260924 19:31:51.824263 5053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:31:51.828857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37929
--builtin_ntp_servers=127.0.98.20:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:31:51.908999 5064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:51.909214 5064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:51.909243 5064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:51.910384 5064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:51.910422 5064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:51.910457 5064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:51.911764 5064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:51.912000 5064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:51.912274 5064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:51.917053 5069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.917043 5072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:51.917032 5070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:51.917270 5064 server_base.cc:1034] running on GCE node
I20260924 19:31:51.917450 5064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:51.917680 5064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:51.918844 5064 hybrid_clock.cc:648] HybridClock initialized: now 1790278311918830 us; error 30 us; skew 500 ppm
I20260924 19:31:51.920007 5064 webserver.cc:460] Webserver started at http://127.0.98.1:45281/ using document root <none> and password file <none>
I20260924 19:31:51.920233 5064 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:51.920282 5064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:51.920408 5064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:51.921553 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "94b21dc52df6431b8d34d0d01f1807b9"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "09fb8312a847ea2aa5862239a948b43f"
server_key_iv: "82e4ce2a91c302050478dd91838c777e"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.921890 5064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "94b21dc52df6431b8d34d0d01f1807b9"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "09fb8312a847ea2aa5862239a948b43f"
server_key_iv: "82e4ce2a91c302050478dd91838c777e"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.923177 5064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:31:51.923914 5078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.924090 5064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:51.924176 5064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "94b21dc52df6431b8d34d0d01f1807b9"
format_stamp: "Formatted at 2026-09-24 19:31:51 on dist-test-slave-m8jb"
server_key: "09fb8312a847ea2aa5862239a948b43f"
server_key_iv: "82e4ce2a91c302050478dd91838c777e"
server_key_version: "encryptionkey@0"
I20260924 19:31:51.924247 5064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:51.927611 5064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:51.927865 5064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:51.927987 5064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:51.928177 5064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:51.928452 5064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:51.928483 5064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.928508 5064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:51.928525 5064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:51.933822 5064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34337
I20260924 19:31:51.933887 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34337 every 8 connection(s)
I20260924 19:31:51.934158 5064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:31:51.935302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 5064
I20260924 19:31:51.935395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:51.935542 392 external_mini_cluster.cc:1428] Setting key 23d1a938826dc0008fac081383629e15
I20260924 19:31:51.936462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37929
--builtin_ntp_servers=127.0.98.20:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 19:31:51.937096 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.934806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34639 (local address 127.0.98.62:37929)
0924 19:31:51.934869 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:51.934871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:51.935086 (+ 215us) server_negotiation.cc:407] Connection header received
0924 19:31:51.935597 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:51.935598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:51.935606 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:51.935670 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:51.936857 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:51.936858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:51.936859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:51.936863 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:51.936878 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:51.936883 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:51.936930 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:51.937014 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 19:31:51.937027 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:31:51.937285 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:51.934782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37929 (local address 127.0.98.1:34639)
0924 19:31:51.934966 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:51.934999 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:51.935505 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:51.935741 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:51.935746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:51.935756 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:51.936706 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:31:51.936707 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:51.936743 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:31:51.936777 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:51.936943 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:51.936947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:51.936950 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:51.937024 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:31:51.937083 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260924 19:31:51.937820 5192 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37929
I20260924 19:31:51.937906 5192 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:51.938035 5192 heartbeater.cc:511] Master 127.0.98.62:37929 requested a full tablet report, sending...
I20260924 19:31:51.938530 5004 ts_manager.cc:194] Registered new tserver with Master: 94b21dc52df6431b8d34d0d01f1807b9 (127.0.98.1:34337)
W20260924 19:31:52.015034 5195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:52.015240 5195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:52.015261 5195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:52.016366 5195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:52.016399 5195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:52.016423 5195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:52.017740 5195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:52.017948 5195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:52.018208 5195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:52.023133 5201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:52.023263 5195 server_base.cc:1034] running on GCE node
W20260924 19:31:52.023135 5203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:52.023203 5200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:52.023588 5195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:52.023823 5195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:52.024969 5195 hybrid_clock.cc:648] HybridClock initialized: now 1790278312024957 us; error 30 us; skew 500 ppm
I20260924 19:31:52.026160 5195 webserver.cc:460] Webserver started at http://127.0.98.2:43433/ using document root <none> and password file <none>
I20260924 19:31:52.026391 5195 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:52.026439 5195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:52.026563 5195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:52.027673 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "370e090565d34966b51016c6271e45f8"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "b6ba91e8acc26103fcc38ff42aa83b77"
server_key_iv: "ea1cfbc865db366ed97a0d3729a1be5f"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.028039 5195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "370e090565d34966b51016c6271e45f8"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "b6ba91e8acc26103fcc38ff42aa83b77"
server_key_iv: "ea1cfbc865db366ed97a0d3729a1be5f"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.029337 5195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:52.030009 5209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.030192 5195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:52.030289 5195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "370e090565d34966b51016c6271e45f8"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "b6ba91e8acc26103fcc38ff42aa83b77"
server_key_iv: "ea1cfbc865db366ed97a0d3729a1be5f"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.030367 5195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:52.033677 5195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:52.033929 5195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:52.034047 5195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:52.034235 5195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:52.034520 5195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:52.034552 5195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.034578 5195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:52.034595 5195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.039809 5195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41035
I20260924 19:31:52.039873 5322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41035 every 8 connection(s)
I20260924 19:31:52.040143 5195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:31:52.040745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 5195
I20260924 19:31:52.040835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:52.041007 392 external_mini_cluster.cc:1428] Setting key 9c90bbc286e84b29d6e9a5de0082115d
I20260924 19:31:52.041885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37929
--builtin_ntp_servers=127.0.98.20:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 19:31:52.043219 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.040849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53081 (local address 127.0.98.62:37929)
0924 19:31:52.040915 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.040917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.041207 (+ 290us) server_negotiation.cc:407] Connection header received
0924 19:31:52.041744 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.041744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.041752 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.041834 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.042976 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.042978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.042978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.042983 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.043002 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.043007 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.043070 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.043155 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.043166 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:31:52.043431 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.040820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37929 (local address 127.0.98.2:53081)
0924 19:31:52.041091 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.041130 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.041657 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.041894 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.041897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.041905 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:52.042798 (+ 893us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:31:52.042799 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:52.042826 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:31:52.042870 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:52.043075 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:52.043080 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:52.043083 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:52.043159 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 19:31:52.043224 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260924 19:31:52.043867 5323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37929
I20260924 19:31:52.043952 5323 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:52.044083 5323 heartbeater.cc:511] Master 127.0.98.62:37929 requested a full tablet report, sending...
I20260924 19:31:52.044425 5004 ts_manager.cc:194] Registered new tserver with Master: 370e090565d34966b51016c6271e45f8 (127.0.98.2:41035)
W20260924 19:31:52.121567 5326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:52.121773 5326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:52.121800 5326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:52.122987 5326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:52.123028 5326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:52.123061 5326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:52.124413 5326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37929
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:52.124688 5326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:52.124984 5326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:52.129788 5331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:52.129793 5334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:52.129799 5332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:52.129992 5326 server_base.cc:1034] running on GCE node
I20260924 19:31:52.130179 5326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:52.130401 5326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:52.131577 5326 hybrid_clock.cc:648] HybridClock initialized: now 1790278312131554 us; error 35 us; skew 500 ppm
I20260924 19:31:52.132867 5326 webserver.cc:460] Webserver started at http://127.0.98.3:39355/ using document root <none> and password file <none>
I20260924 19:31:52.133105 5326 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:52.133153 5326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:52.133281 5326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:52.134351 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "643f86c5e26345479d01bd10e4653e34"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "20254e449e6eeb0ff4a9aa9ff80240aa"
server_key_iv: "51e8b390b7d2170296c5a9345ea32dd4"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.134684 5326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "643f86c5e26345479d01bd10e4653e34"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "20254e449e6eeb0ff4a9aa9ff80240aa"
server_key_iv: "51e8b390b7d2170296c5a9345ea32dd4"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.135932 5326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:52.136685 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.136866 5326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:52.136947 5326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "643f86c5e26345479d01bd10e4653e34"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "20254e449e6eeb0ff4a9aa9ff80240aa"
server_key_iv: "51e8b390b7d2170296c5a9345ea32dd4"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.137015 5326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:52.140255 5326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:52.140512 5326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:52.140661 5326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:52.140870 5326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:52.141151 5326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:52.141183 5326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.141209 5326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:52.141227 5326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.146562 5326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43635
I20260924 19:31:52.146613 5453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43635 every 8 connection(s)
I20260924 19:31:52.146926 5326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:31:52.149719 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.147606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47711 (local address 127.0.98.62:37929)
0924 19:31:52.147653 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.147655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.147869 (+ 214us) server_negotiation.cc:407] Connection header received
0924 19:31:52.148309 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.148310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.148316 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.148391 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.149511 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.149513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.149514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.149518 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.149533 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.149537 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.149596 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.149646 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.149658 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:31:52.149881 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.147578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37929 (local address 127.0.98.3:47711)
0924 19:31:52.147741 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.147782 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.148230 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.148450 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.148453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.148462 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:52.149345 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:31:52.149346 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:52.149375 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:31:52.149404 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:52.149597 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:52.149599 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:52.149602 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:52.149648 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:31:52.149696 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260924 19:31:52.150352 5454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37929
I20260924 19:31:52.150436 5454 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:52.150578 5454 heartbeater.cc:511] Master 127.0.98.62:37929 requested a full tablet report, sending...
I20260924 19:31:52.150986 5004 ts_manager.cc:194] Registered new tserver with Master: 643f86c5e26345479d01bd10e4653e34 (127.0.98.3:43635)
I20260924 19:31:52.156741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 5326
I20260924 19:31:52.156844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:52.157015 392 external_mini_cluster.cc:1428] Setting key 0a0f646eb444c125de8380b5d2286a80
I20260924 19:31:52.157864 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:31:52.160200 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.159142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.0.98.62:37929)
0924 19:31:52.159204 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.159206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.159319 (+ 113us) server_negotiation.cc:407] Connection header received
0924 19:31:52.159380 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.159381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.159387 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.159432 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.159978 (+ 546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.159979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.159980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.159984 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.159998 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.160002 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.160048 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.160109 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.160121 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:31:52.161762 5004 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:52.163300 5004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:52.166937 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.165440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43635 (local address 127.0.0.1:47324)
0924 19:31:52.165739 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.165749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.165793 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.165978 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.165985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.165991 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:52.166474 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:31:52.166474 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:52.166492 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:31:52.166517 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:52.166780 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:52.166786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:52.166790 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:52.166856 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:31:52.166869 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":88,"threads_started":1}
I20260924 19:31:52.166973 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.165535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.0.98.3:43635)
0924 19:31:52.165852 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.165857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.165868 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:31:52.165894 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.165896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.165904 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.165977 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.166592 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.166600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.166600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.166604 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.166638 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.166704 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.166768 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.166865 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.166882 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":82,"threads_started":1}
I20260924 19:31:52.167251 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.165398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41035 (local address 127.0.0.1:43996)
0924 19:31:52.165584 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.165627 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.165679 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.166166 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.166167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.166180 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:52.166642 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:31:52.166643 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:52.166649 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:31:52.166661 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:52.167132 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:52.167133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:52.167134 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:31:52.167180 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:31:52.167191 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":62,"threads_started":1}
I20260924 19:31:52.167438 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.165557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.0.98.2:41035)
0924 19:31:52.165740 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.165744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.165755 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:31:52.165783 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.165785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.165794 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.165859 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.166759 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.166771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.166771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.166776 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.166813 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.166844 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.166895 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.167346 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.167363 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260924 19:31:52.167613 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.165616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34337 (local address 127.0.0.1:58676)
0924 19:31:52.166337 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.166345 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.166389 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.166784 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.166785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.166788 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:52.167164 (+ 376us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:31:52.167164 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:52.167188 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:31:52.167199 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:52.167477 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:52.167478 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:52.167479 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:31:52.167542 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 19:31:52.167553 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":79,"threads_started":1}
I20260924 19:31:52.167639 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.165805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.0.98.1:34337)
0924 19:31:52.165981 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.165986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.166393 (+ 407us) server_negotiation.cc:407] Connection header received
0924 19:31:52.166431 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.166434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.166443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.166516 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.167271 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.167279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.167279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.167283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.167380 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.167409 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.167463 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.167551 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.167568 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260924 19:31:52.167694 5388 tablet_service.cc:1467] Processing CreateTablet for tablet f32c0b97aa2844c4893701f01b17b25a (DEFAULT_TABLE table=test-table [id=eaac4eed3afc4f0fbf61bc926a3878ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:52.168001 5388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32c0b97aa2844c4893701f01b17b25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:52.167971 5257 tablet_service.cc:1467] Processing CreateTablet for tablet f32c0b97aa2844c4893701f01b17b25a (DEFAULT_TABLE table=test-table [id=eaac4eed3afc4f0fbf61bc926a3878ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:52.168205 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32c0b97aa2844c4893701f01b17b25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:52.170485 5468 tablet_bootstrap.cc:492] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: Bootstrap starting.
I20260924 19:31:52.170765 5469 tablet_bootstrap.cc:492] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: Bootstrap starting.
I20260924 19:31:52.170984 5126 tablet_service.cc:1467] Processing CreateTablet for tablet f32c0b97aa2844c4893701f01b17b25a (DEFAULT_TABLE table=test-table [id=eaac4eed3afc4f0fbf61bc926a3878ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:52.171212 5126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f32c0b97aa2844c4893701f01b17b25a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:52.171204 5468 tablet_bootstrap.cc:654] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:52.171340 5469 tablet_bootstrap.cc:654] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:52.171509 5468 log.cc:824] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:52.171638 5469 log.cc:824] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:52.172232 5468 tablet_bootstrap.cc:492] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: No bootstrap required, opened a new log
I20260924 19:31:52.172276 5468 ts_tablet_manager.cc:1397] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:52.172312 5469 tablet_bootstrap.cc:492] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: No bootstrap required, opened a new log
I20260924 19:31:52.172380 5469 ts_tablet_manager.cc:1397] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:52.174131 5472 tablet_bootstrap.cc:492] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9: Bootstrap starting.
I20260924 19:31:52.174182 5468 raft_consensus.cc:348] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.174330 5468 raft_consensus.cc:374] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:52.174361 5468 raft_consensus.cc:729] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643f86c5e26345479d01bd10e4653e34, State: Initialized, Role: FOLLOWER
I20260924 19:31:52.174474 5468 consensus_queue.cc:260] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.174708 5468 ts_tablet_manager.cc:1428] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:31:52.174773 5472 tablet_bootstrap.cc:654] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:52.174845 5454 heartbeater.cc:503] Master 127.0.98.62:37929 was elected leader, sending a full tablet report...
I20260924 19:31:52.174841 5469 raft_consensus.cc:348] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.174997 5469 raft_consensus.cc:374] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:52.175036 5469 raft_consensus.cc:729] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370e090565d34966b51016c6271e45f8, State: Initialized, Role: FOLLOWER
I20260924 19:31:52.175076 5472 log.cc:824] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:52.175146 5469 consensus_queue.cc:260] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.175380 5469 ts_tablet_manager.cc:1428] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:31:52.175432 5323 heartbeater.cc:503] Master 127.0.98.62:37929 was elected leader, sending a full tablet report...
I20260924 19:31:52.176039 5472 tablet_bootstrap.cc:492] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9: No bootstrap required, opened a new log
I20260924 19:31:52.176096 5472 ts_tablet_manager.cc:1397] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:52.177979 5472 raft_consensus.cc:348] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.178097 5472 raft_consensus.cc:374] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:52.178126 5472 raft_consensus.cc:729] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94b21dc52df6431b8d34d0d01f1807b9, State: Initialized, Role: FOLLOWER
I20260924 19:31:52.178224 5472 consensus_queue.cc:260] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.178436 5472 ts_tablet_manager.cc:1428] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:31:52.178479 5192 heartbeater.cc:503] Master 127.0.98.62:37929 was elected leader, sending a full tablet report...
W20260924 19:31:52.184751 5193 tablet.cc:2369] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:31:52.221453 5475 raft_consensus.cc:482] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:52.221576 5475 raft_consensus.cc:504] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.221925 5475 leader_election.cc:288] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 643f86c5e26345479d01bd10e4653e34 (127.0.98.3:43635), 94b21dc52df6431b8d34d0d01f1807b9 (127.0.98.1:34337)
I20260924 19:31:52.223409 5325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.222187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43635 (local address 127.0.98.2:36971)
0924 19:31:52.222324 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.222338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.222400 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.222558 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.222558 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.222561 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:52.223061 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:31:52.223062 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:52.223071 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:31:52.223085 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:52.223262 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:52.223263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:52.223264 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:31:52.223314 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:31:52.223327 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 19:31:52.223409 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.222309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36971 (local address 127.0.98.3:43635)
0924 19:31:52.222383 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.222385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.222396 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:31:52.222488 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.222489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.222494 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.222542 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.223165 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.223167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.223167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.223172 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.223190 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.223195 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.223257 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.223323 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.223338 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:31:52.223695 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.222411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41819 (local address 127.0.98.1:34337)
0924 19:31:52.222490 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.222492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.222671 (+ 179us) server_negotiation.cc:407] Connection header received
0924 19:31:52.222702 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.222703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.222709 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.222761 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.223409 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.223410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.223410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.223415 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.223433 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.223437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.223497 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.223609 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.223624 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:31:52.223875 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.222305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34337 (local address 127.0.98.2:41819)
0924 19:31:52.222543 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.222551 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.222609 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.222775 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.222775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.222778 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:52.223312 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:31:52.223313 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:52.223320 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:31:52.223331 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:52.223526 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:52.223527 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:52.223528 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:31:52.223780 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 19:31:52.223796 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":123,"threads_started":1}
I20260924 19:31:52.223814 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32c0b97aa2844c4893701f01b17b25a" candidate_uuid: "370e090565d34966b51016c6271e45f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643f86c5e26345479d01bd10e4653e34" is_pre_election: true
I20260924 19:31:52.223980 5408 raft_consensus.cc:2454] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 370e090565d34966b51016c6271e45f8 in term 0.
I20260924 19:31:52.224224 5212 leader_election.cc:302] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370e090565d34966b51016c6271e45f8, 643f86c5e26345479d01bd10e4653e34; no voters:
I20260924 19:31:52.224213 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32c0b97aa2844c4893701f01b17b25a" candidate_uuid: "370e090565d34966b51016c6271e45f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b21dc52df6431b8d34d0d01f1807b9" is_pre_election: true
I20260924 19:31:52.224339 5146 raft_consensus.cc:2454] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 370e090565d34966b51016c6271e45f8 in term 0.
I20260924 19:31:52.224359 5475 raft_consensus.cc:2781] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:52.224431 5475 raft_consensus.cc:482] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:52.224466 5475 raft_consensus.cc:3037] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:52.225275 5475 raft_consensus.cc:504] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.225394 5475 leader_election.cc:288] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [CANDIDATE]: Term 1 election: Requested vote from peers 643f86c5e26345479d01bd10e4653e34 (127.0.98.3:43635), 94b21dc52df6431b8d34d0d01f1807b9 (127.0.98.1:34337)
I20260924 19:31:52.225574 5408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32c0b97aa2844c4893701f01b17b25a" candidate_uuid: "370e090565d34966b51016c6271e45f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643f86c5e26345479d01bd10e4653e34"
I20260924 19:31:52.225641 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f32c0b97aa2844c4893701f01b17b25a" candidate_uuid: "370e090565d34966b51016c6271e45f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94b21dc52df6431b8d34d0d01f1807b9"
I20260924 19:31:52.225663 5408 raft_consensus.cc:3037] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:52.225729 5146 raft_consensus.cc:3037] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:52.226419 5146 raft_consensus.cc:2454] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 370e090565d34966b51016c6271e45f8 in term 1.
I20260924 19:31:52.226438 5408 raft_consensus.cc:2454] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 370e090565d34966b51016c6271e45f8 in term 1.
I20260924 19:31:52.226569 5212 leader_election.cc:302] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370e090565d34966b51016c6271e45f8, 643f86c5e26345479d01bd10e4653e34; no voters:
I20260924 19:31:52.226691 5475 raft_consensus.cc:2781] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:52.226858 5475 raft_consensus.cc:686] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 1 LEADER]: Becoming Leader. State: Replica: 370e090565d34966b51016c6271e45f8, State: Running, Role: LEADER
I20260924 19:31:52.226944 5475 consensus_queue.cc:237] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } }
I20260924 19:31:52.227878 5003 catalog_manager.cc:5495] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 370e090565d34966b51016c6271e45f8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "370e090565d34966b51016c6271e45f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370e090565d34966b51016c6271e45f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 } health_report { overall_health: UNKNOWN } } }
I20260924 19:31:52.241173 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.240207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.0.98.2:41035)
0924 19:31:52.240263 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.240265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.240273 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:52.240313 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.240314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.240325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.240373 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.240997 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.240998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.240998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.241002 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.241016 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.241020 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.241063 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.241092 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.241102 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:31:52.242935 5146 raft_consensus.cc:1261] T f32c0b97aa2844c4893701f01b17b25a P 94b21dc52df6431b8d34d0d01f1807b9 [term 1 FOLLOWER]: Refusing update from remote peer 370e090565d34966b51016c6271e45f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:52.242955 5408 raft_consensus.cc:1261] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 1 FOLLOWER]: Refusing update from remote peer 370e090565d34966b51016c6271e45f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:52.243299 5475 consensus_queue.cc:1035] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94b21dc52df6431b8d34d0d01f1807b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:52.243448 5478 consensus_queue.cc:1035] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643f86c5e26345479d01bd10e4653e34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:52.245689 5481 mvcc.cc:205] Tried to move back new op lower bound from 7332979966944403456 to 7332979966882488320. Current Snapshot: MvccSnapshot[applied={T|T < 7332979966944403456}]
I20260924 19:31:52.245697 5482 mvcc.cc:205] Tried to move back new op lower bound from 7332979966944403456 to 7332979966882488320. Current Snapshot: MvccSnapshot[applied={T|T < 7332979966944403456}]
I20260924 19:31:52.247984 5003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:52.248083 5003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:52.248835 5003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353334607dd34c3584aa00f20367a91f: Sending DeleteTablet for 3 replicas of tablet f32c0b97aa2844c4893701f01b17b25a
I20260924 19:31:52.249136 5257 tablet_service.cc:1514] Processing DeleteTablet for tablet f32c0b97aa2844c4893701f01b17b25a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:52 UTC) from {username='slave'} at 127.0.0.1:43996
I20260924 19:31:52.249226 5388 tablet_service.cc:1514] Processing DeleteTablet for tablet f32c0b97aa2844c4893701f01b17b25a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:52 UTC) from {username='slave'} at 127.0.0.1:47324
I20260924 19:31:52.249406 5491 tablet_replica.cc:330] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: stopping tablet replica
I20260924 19:31:52.249428 5126 tablet_service.cc:1514] Processing DeleteTablet for tablet f32c0b97aa2844c4893701f01b17b25a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:52 UTC) from {username='slave'} at 127.0.0.1:58676
I20260924 19:31:52.249497 5491 raft_consensus.cc:2229] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:31:52.249560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 5064
I20260924 19:31:52.249615 5491 raft_consensus.cc:2258] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:52.251693 5491 ts_tablet_manager.cc:1905] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:52.253600 5491 ts_tablet_manager.cc:1918] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:52.253675 5491 log.cc:1196] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/f32c0b97aa2844c4893701f01b17b25a
I20260924 19:31:52.253932 5491 ts_tablet_manager.cc:1939] T f32c0b97aa2844c4893701f01b17b25a P 370e090565d34966b51016c6271e45f8: Deleting consensus metadata
W20260924 19:31:52.254256 4990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34337 (error 108)
I20260924 19:31:52.254374 4988 catalog_manager.cc:4841] TS 370e090565d34966b51016c6271e45f8 (127.0.98.2:41035): tablet f32c0b97aa2844c4893701f01b17b25a (table test-table [id=eaac4eed3afc4f0fbf61bc926a3878ce]) successfully deleted
I20260924 19:31:52.254479 5492 tablet_replica.cc:330] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: stopping tablet replica
I20260924 19:31:52.254592 5492 raft_consensus.cc:2229] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:52.254714 5492 raft_consensus.cc:2258] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:52.254885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 5195
I20260924 19:31:52.254887 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.254745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34337 (local address 127.0.0.1:58684)
0924 19:31:52.254811 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.254855 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 19:31:52.255000 4990 catalog_manager.cc:4584] TS 94b21dc52df6431b8d34d0d01f1807b9 (127.0.98.1:34337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f32c0b97aa2844c4893701f01b17b25a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34337: connect: Connection refused (error 111)
I20260924 19:31:52.255865 5492 ts_tablet_manager.cc:1905] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:52.257407 5492 ts_tablet_manager.cc:1918] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:52.257469 5492 log.cc:1196] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/f32c0b97aa2844c4893701f01b17b25a
I20260924 19:31:52.257663 5492 ts_tablet_manager.cc:1939] T f32c0b97aa2844c4893701f01b17b25a P 643f86c5e26345479d01bd10e4653e34: Deleting consensus metadata
I20260924 19:31:52.258082 4990 catalog_manager.cc:4841] TS 643f86c5e26345479d01bd10e4653e34 (127.0.98.3:43635): tablet f32c0b97aa2844c4893701f01b17b25a (table test-table [id=eaac4eed3afc4f0fbf61bc926a3878ce]) successfully deleted
I20260924 19:31:52.259967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 5326
I20260924 19:31:52.263612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 4973
2026-09-24T19:31:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (589 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 19:31:52.281599 392 mini_oidc.cc:152] Starting JWKS server
I20260924 19:31:52.282636 392 webserver.cc:460] Webserver started at https://127.0.0.1:44969/ using document root <none> and password file <none>
I20260924 19:31:52.282727 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:31:52.282894 392 webserver.cc:460] Webserver started at http://127.0.0.1:37181/ using document root <none> and password file <none>
2026-09-24T19:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:52Z Disabled control of system clock
I20260924 19:31:52.290033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34935
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34935
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44969/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/testchainca.cert with env {}
W20260924 19:31:52.367430 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:52.367632 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:52.367660 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:52.368817 5500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:52.368854 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:52.368865 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:52.368876 5500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:52.368886 5500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:31:52.368896 5500 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:31:52.368906 5500 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44969/jwks.json
I20260924 19:31:52.370373 5500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44969/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:52.370569 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:52.370745 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:52.375404 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:52.375387 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:52.375407 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:52.375517 5500 server_base.cc:1034] running on GCE node
I20260924 19:31:52.375928 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:52.376200 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:52.377347 5500 hybrid_clock.cc:648] HybridClock initialized: now 1790278312377328 us; error 29 us; skew 500 ppm
I20260924 19:31:52.378575 5500 webserver.cc:460] Webserver started at http://127.0.98.62:35601/ using document root <none> and password file <none>
I20260924 19:31:52.378803 5500 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:52.378850 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:52.378970 5500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:52.379979 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "8c36fc5e835245e799a2d00681a0ffa7"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "50c3847c844f9f5507ff5e37ec440c77"
server_key_iv: "d4ffa714b0943d48fcb857795be5778e"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.380307 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c36fc5e835245e799a2d00681a0ffa7"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "50c3847c844f9f5507ff5e37ec440c77"
server_key_iv: "d4ffa714b0943d48fcb857795be5778e"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.381587 5500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:52.382364 5514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.382539 5500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:52.382630 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "8c36fc5e835245e799a2d00681a0ffa7"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "50c3847c844f9f5507ff5e37ec440c77"
server_key_iv: "d4ffa714b0943d48fcb857795be5778e"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.382699 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:52.401890 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:52.402215 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:52.402355 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:52.406090 5500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34935
I20260924 19:31:52.406132 5566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34935 every 8 connection(s)
I20260924 19:31:52.406467 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:52.407394 5567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:52.409669 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7: Bootstrap starting.
I20260924 19:31:52.410238 5567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:52.410498 5567 log.cc:824] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:52.411093 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7: No bootstrap required, opened a new log
I20260924 19:31:52.412611 5567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c36fc5e835245e799a2d00681a0ffa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } }
I20260924 19:31:52.412774 5567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:52.412798 5567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c36fc5e835245e799a2d00681a0ffa7, State: Initialized, Role: FOLLOWER
I20260924 19:31:52.412945 5567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c36fc5e835245e799a2d00681a0ffa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } }
I20260924 19:31:52.413007 5567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:52.413028 5567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:52.413074 5567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:52.413558 5567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c36fc5e835245e799a2d00681a0ffa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } }
I20260924 19:31:52.413642 5567 leader_election.cc:302] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c36fc5e835245e799a2d00681a0ffa7; no voters:
I20260924 19:31:52.413793 5567 leader_election.cc:288] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:52.413856 5570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:52.414023 5567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:52.414069 5570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c36fc5e835245e799a2d00681a0ffa7, State: Running, Role: LEADER
I20260924 19:31:52.414158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 5500
I20260924 19:31:52.414165 5570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c36fc5e835245e799a2d00681a0ffa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } }
I20260924 19:31:52.414275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:52.414469 392 external_mini_cluster.cc:1428] Setting key 7ae9ae56ae65b57f2dd5741dc66e265d
I20260924 19:31:52.414661 5571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c36fc5e835245e799a2d00681a0ffa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c36fc5e835245e799a2d00681a0ffa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } } }
I20260924 19:31:52.414740 5571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:52.414955 5579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:52.415484 5579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:52.415661 5572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c36fc5e835245e799a2d00681a0ffa7. Latest consensus state: current_term: 1 leader_uuid: "8c36fc5e835245e799a2d00681a0ffa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c36fc5e835245e799a2d00681a0ffa7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34935 } } }
I20260924 19:31:52.415823 5572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:52.417220 5579 catalog_manager.cc:1347] Generated new cluster ID: 4e0cc13ca26e4d8dae168bfdb8b22332
I20260924 19:31:52.417271 5579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:52.424006 5579 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:52.424623 5579 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:52.429132 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.415339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.0.98.62:34935)
0924 19:31:52.423816 (+ 8477us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.423826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.423866 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:31:52.423987 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.423995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.424070 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.424232 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.424469 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.425837 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.426987 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.427608 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.427719 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.428394 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.428398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.428399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.428599 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.428667 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.428682 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.428738 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.428814 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.428905 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8397,"thread_start_us":83,"threads_started":1}
I20260924 19:31:52.429939 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:31:52.435600 5579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c36fc5e835245e799a2d00681a0ffa7: Generated new TSK 0
I20260924 19:31:52.435753 5579 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:31:52.460345 5597 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 19:31:52.460419 5597 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:31:52.460716 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.445646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.0.98.62:34935)
0924 19:31:52.445694 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.445696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.445785 (+ 89us) server_negotiation.cc:407] Connection header received
0924 19:31:52.445847 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.445848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.445858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.445929 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:31:52.446045 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.446820 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.447531 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.447842 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.447906 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.448053 (+ 147us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:31:52.460481 (+ 12428us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:31:52.460618 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 19:31:52.460768 5588 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60442: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 19:31:52.460996 5596 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:31:52.445572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34935 (local address 127.0.0.1:60442)
0924 19:31:52.445722 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.445732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.445784 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.445933 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.445934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.445935 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:31:52.445992 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:52.445994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.446883 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.446883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:52.447464 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:52.447465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.447902 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.447905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:52.447944 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.447959 (+ 15us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:31:52.460583 (+ 12624us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:31:52.460699 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 19:31:52.461485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 5500
2026-09-24T19:31:52Z chronyd exiting
I20260924 19:31:52.496670 392 test_util.cc:175] -----------------------------------------------
I20260924 19:31:52.496757 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790278290121024-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 19:31:52.497495 392 mini_oidc.cc:152] Starting JWKS server
I20260924 19:31:52.498176 392 webserver.cc:460] Webserver started at https://127.0.0.1:45717/ using document root <none> and password file <none>
I20260924 19:31:52.498217 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:31:52.498363 392 webserver.cc:460] Webserver started at http://127.0.0.1:44283/ using document root <none> and password file <none>
2026-09-24T19:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:52Z Disabled control of system clock
I20260924 19:31:52.505558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32931
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32931
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45717/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/testchainca.cert with env {}
W20260924 19:31:52.584165 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:52.584354 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:52.584375 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:52.585562 5603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:52.585604 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:52.585618 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:52.585628 5603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:52.585646 5603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:31:52.585664 5603 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:31:52.585673 5603 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45717/jwks.json
I20260924 19:31:52.587105 5603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45717/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:52.587281 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:52.587437 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:52.591902 5608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:52.591908 5611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:52.592069 5603 server_base.cc:1034] running on GCE node
W20260924 19:31:52.591910 5609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:52.592365 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:52.592610 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:52.593788 5603 hybrid_clock.cc:648] HybridClock initialized: now 1790278312593776 us; error 29 us; skew 500 ppm
I20260924 19:31:52.595013 5603 webserver.cc:460] Webserver started at http://127.0.98.62:39829/ using document root <none> and password file <none>
I20260924 19:31:52.595244 5603 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:52.595291 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:52.595403 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:52.596401 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "9df1270a826647d99c0280ad7cc9b61c"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "ae65e9a84ff9fc2d213a0d4a12e299c5"
server_key_iv: "c8257e1b525d51bc161d408664294a51"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.596772 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "9df1270a826647d99c0280ad7cc9b61c"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "ae65e9a84ff9fc2d213a0d4a12e299c5"
server_key_iv: "c8257e1b525d51bc161d408664294a51"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.598098 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:52.598834 5617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.599020 5603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:52.599100 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "9df1270a826647d99c0280ad7cc9b61c"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "ae65e9a84ff9fc2d213a0d4a12e299c5"
server_key_iv: "c8257e1b525d51bc161d408664294a51"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.599162 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:52.609019 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:52.609275 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:52.609383 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:52.613022 5603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32931
I20260924 19:31:52.613058 5669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32931 every 8 connection(s)
I20260924 19:31:52.613399 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:52.614245 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:52.616348 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c: Bootstrap starting.
I20260924 19:31:52.616922 5670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:52.617136 5670 log.cc:824] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:52.617748 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c: No bootstrap required, opened a new log
I20260924 19:31:52.619298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 5603
I20260924 19:31:52.619241 5670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df1270a826647d99c0280ad7cc9b61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32931 } }
I20260924 19:31:52.619359 5670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:52.619377 5670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df1270a826647d99c0280ad7cc9b61c, State: Initialized, Role: FOLLOWER
I20260924 19:31:52.619391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:52.619520 5670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df1270a826647d99c0280ad7cc9b61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32931 } }
I20260924 19:31:52.619565 392 external_mini_cluster.cc:1428] Setting key 844fc38265d3d6070b10276038c8b3ef
I20260924 19:31:52.619592 5670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:52.619629 5670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:52.619685 5670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:52.620323 5670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df1270a826647d99c0280ad7cc9b61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32931 } }
I20260924 19:31:52.620421 5670 leader_election.cc:302] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9df1270a826647d99c0280ad7cc9b61c; no voters:
I20260924 19:31:52.620568 5670 leader_election.cc:288] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:52.620594 5675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:52.620757 5675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [term 1 LEADER]: Becoming Leader. State: Replica: 9df1270a826647d99c0280ad7cc9b61c, State: Running, Role: LEADER
I20260924 19:31:52.620826 5670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:52.620848 5675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df1270a826647d99c0280ad7cc9b61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32931 } }
I20260924 19:31:52.621290 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9df1270a826647d99c0280ad7cc9b61c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df1270a826647d99c0280ad7cc9b61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32931 } } }
I20260924 19:31:52.621394 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:52.621685 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9df1270a826647d99c0280ad7cc9b61c. Latest consensus state: current_term: 1 leader_uuid: "9df1270a826647d99c0280ad7cc9b61c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9df1270a826647d99c0280ad7cc9b61c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32931 } } }
I20260924 19:31:52.621744 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c [sys.catalog]: This master's current role is: LEADER
W20260924 19:31:52.622421 5691 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:31:52.622469 5691 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:31:52.622524 5684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:52.622660 5684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:52.624192 5684 catalog_manager.cc:1347] Generated new cluster ID: 6e7d49f66d2c4aeea2098529c443bb0b
I20260924 19:31:52.624234 5684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:52.625996 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.619995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.0.98.62:32931)
0924 19:31:52.620233 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.620242 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.620276 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:31:52.620389 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.620396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.620479 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.620622 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.621085 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.622550 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.623418 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.624043 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.624160 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.625060 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.625064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.625064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.625291 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.625369 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.625387 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.625450 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.625702 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.625786 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260924 19:31:52.626865 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:31:52.637782 5684 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:52.638264 5684 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:52.650274 5684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9df1270a826647d99c0280ad7cc9b61c: Generated new TSK 0
I20260924 19:31:52.650455 5684 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:31:52.672353 5700 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 19:31:52.672429 5700 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:31:52.672838 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.657823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.98.62:32931)
0924 19:31:52.657878 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.657880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.657979 (+ 99us) server_negotiation.cc:407] Connection header received
0924 19:31:52.658032 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.658033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.658045 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.658121 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:31:52.658247 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.659048 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.659859 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.660182 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.660249 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.660450 (+ 201us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:31:52.672573 (+ 12123us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:31:52.672769 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 19:31:52.672901 5674 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 19:31:52.672896 5699 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:31:52.657738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32931 (local address 127.0.0.1:35290)
0924 19:31:52.657915 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.657926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.657979 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.658141 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.658141 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.658143 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:31:52.658193 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:52.658195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.659153 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.659154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:52.659777 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:52.659780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.660285 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.660289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:52.660342 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.660347 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:31:52.672719 (+ 12372us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:31:52.672821 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45717/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260924 19:31:52.673259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 5603
2026-09-24T19:31:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 19:31:52.709928 392 mini_oidc.cc:152] Starting JWKS server
I20260924 19:31:52.710613 392 webserver.cc:460] Webserver started at https://127.0.0.1:43397/ using document root <none> and password file <none>
I20260924 19:31:52.710659 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:31:52.710803 392 webserver.cc:460] Webserver started at http://127.0.0.1:38211/ using document root <none> and password file <none>
2026-09-24T19:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:52Z Disabled control of system clock
I20260924 19:31:52.717495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43343
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43343
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43397/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:31:52.794770 5706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:52.794981 5706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:52.795012 5706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:52.796168 5706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:52.796208 5706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:52.796224 5706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:52.796239 5706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:52.796253 5706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:31:52.796267 5706 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:31:52.796283 5706 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43397/jwks.json
I20260924 19:31:52.797665 5706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43397/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:52.797907 5706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:52.798170 5706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:52.802836 5711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:52.802836 5714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:52.802845 5712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:52.803128 5706 server_base.cc:1034] running on GCE node
I20260924 19:31:52.803391 5706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:52.803658 5706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:52.804836 5706 hybrid_clock.cc:648] HybridClock initialized: now 1790278312804812 us; error 36 us; skew 500 ppm
I20260924 19:31:52.806116 5706 webserver.cc:460] Webserver started at http://127.0.98.62:37529/ using document root <none> and password file <none>
I20260924 19:31:52.806348 5706 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:52.806394 5706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:52.806517 5706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:52.807575 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "1defd4094eec4798800b757ed6eb8ecd"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "cb3c506c2e1ed1c64d7c46b8e066b074"
server_key_iv: "adb148efa5e91ee586c410f63c8b9c68"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.807904 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "1defd4094eec4798800b757ed6eb8ecd"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "cb3c506c2e1ed1c64d7c46b8e066b074"
server_key_iv: "adb148efa5e91ee586c410f63c8b9c68"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.809115 5706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:31:52.809736 5720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:52.809875 5706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:52.809950 5706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "1defd4094eec4798800b757ed6eb8ecd"
format_stamp: "Formatted at 2026-09-24 19:31:52 on dist-test-slave-m8jb"
server_key: "cb3c506c2e1ed1c64d7c46b8e066b074"
server_key_iv: "adb148efa5e91ee586c410f63c8b9c68"
server_key_version: "encryptionkey@0"
I20260924 19:31:52.810015 5706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:52.820362 5706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:52.820616 5706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:52.820778 5706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:52.824287 5706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43343
I20260924 19:31:52.824321 5772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43343 every 8 connection(s)
I20260924 19:31:52.824673 5706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:52.825340 5773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:52.828006 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd: Bootstrap starting.
I20260924 19:31:52.828555 5773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:52.828845 5773 log.cc:824] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:52.829455 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd: No bootstrap required, opened a new log
I20260924 19:31:52.830950 5773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1defd4094eec4798800b757ed6eb8ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260924 19:31:52.831079 5773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:52.831099 5773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1defd4094eec4798800b757ed6eb8ecd, State: Initialized, Role: FOLLOWER
I20260924 19:31:52.831224 5773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1defd4094eec4798800b757ed6eb8ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260924 19:31:52.831290 5773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:52.831321 5773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:52.831360 5773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:52.831430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 5706
I20260924 19:31:52.831517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:52.831681 392 external_mini_cluster.cc:1428] Setting key e1167a460434fbec67566c92ca4c9a5e
I20260924 19:31:52.831923 5773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1defd4094eec4798800b757ed6eb8ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260924 19:31:52.832015 5773 leader_election.cc:302] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1defd4094eec4798800b757ed6eb8ecd; no voters:
I20260924 19:31:52.832187 5773 leader_election.cc:288] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:52.832237 5777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:52.832356 5777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [term 1 LEADER]: Becoming Leader. State: Replica: 1defd4094eec4798800b757ed6eb8ecd, State: Running, Role: LEADER
I20260924 19:31:52.832430 5773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:52.832439 5777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1defd4094eec4798800b757ed6eb8ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } }
I20260924 19:31:52.832806 5777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1defd4094eec4798800b757ed6eb8ecd. Latest consensus state: current_term: 1 leader_uuid: "1defd4094eec4798800b757ed6eb8ecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1defd4094eec4798800b757ed6eb8ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } } }
I20260924 19:31:52.832881 5777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:52.833061 5779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1defd4094eec4798800b757ed6eb8ecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1defd4094eec4798800b757ed6eb8ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43343 } } }
I20260924 19:31:52.833139 5779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:52.833850 5785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:52.834501 5785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:52.835922 5785 catalog_manager.cc:1347] Generated new cluster ID: ba76965352b4422a8c9eb2c9054225be
I20260924 19:31:52.835973 5785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:52.842110 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.832137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.0.98.62:43343)
0924 19:31:52.832365 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.832374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.832408 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:31:52.832519 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.832527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.832597 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.833032 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:52.833130 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.834416 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.835216 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.835796 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.840829 (+ 5033us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.841475 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:52.841479 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:52.841480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:52.841648 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:52.841691 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:52.841704 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:52.841763 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:52.841839 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 19:31:52.841917 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260924 19:31:52.842993 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:31:52.853044 5785 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:52.853579 5785 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:52.875033 5785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1defd4094eec4798800b757ed6eb8ecd: Generated new TSK 0
I20260924 19:31:52.875281 5785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:31:52.889104 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:52.885462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49316 (local address 127.0.98.62:43343)
0924 19:31:52.885506 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:52.885508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:52.885581 (+ 73us) server_negotiation.cc:407] Connection header received
0924 19:31:52.885618 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:52.885619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:52.885631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:52.885711 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:31:52.885832 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.886579 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.887382 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.887682 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.887742 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.887921 (+ 179us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:31:52.888903 (+ 982us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:31:52.889030 (+ 127us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49316: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 19:31:52.889161 5778 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49316: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 19:31:52.889200 5802 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:31:52.885379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43343 (local address 127.0.0.1:49316)
0924 19:31:52.885509 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:52.885519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:52.885571 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:52.885708 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:52.885709 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:52.885710 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:31:52.885765 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:52.885766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.886671 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.886673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:52.887294 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:52.887297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:52.887756 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:52.887761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:52.887799 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:52.887803 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:31:52.889001 (+ 1198us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:31:52.889101 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43343: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43397/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260924 19:31:52.889357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 5706
E20260924 19:31:52.901183 5803 webserver.cc:533] Webserver: sslize failed:
E20260924 19:31:52.901265 5803 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T19:31:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (217 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T19:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:52Z Disabled control of system clock
I20260924 19:31:52.933943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40203
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40203
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/testchainca.cert with env {}
W20260924 19:31:53.011961 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:53.012178 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:53.012199 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:53.013453 5807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:53.013505 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:53.013516 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:53.013527 5807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:53.013537 5807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:31:53.013548 5807 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:31:53.013561 5807 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 19:31:53.015111 5807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:53.015380 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:53.015592 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:53.020404 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:53.020370 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:53.020390 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:53.020606 5807 server_base.cc:1034] running on GCE node
I20260924 19:31:53.020875 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:53.021145 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:53.022279 5807 hybrid_clock.cc:648] HybridClock initialized: now 1790278313022251 us; error 41 us; skew 500 ppm
I20260924 19:31:53.023447 5807 webserver.cc:460] Webserver started at http://127.0.98.62:46813/ using document root <none> and password file <none>
I20260924 19:31:53.023669 5807 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:53.023713 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:53.023854 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:53.024981 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "5a8418dde1c74a8ebfb82f995b77b5f8"
format_stamp: "Formatted at 2026-09-24 19:31:53 on dist-test-slave-m8jb"
server_key: "ebc610c0250b8301183bdf15497bb6a3"
server_key_iv: "0905e0dedda8eca4d390bd1b18fb0f1d"
server_key_version: "encryptionkey@0"
I20260924 19:31:53.025316 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a8418dde1c74a8ebfb82f995b77b5f8"
format_stamp: "Formatted at 2026-09-24 19:31:53 on dist-test-slave-m8jb"
server_key: "ebc610c0250b8301183bdf15497bb6a3"
server_key_iv: "0905e0dedda8eca4d390bd1b18fb0f1d"
server_key_version: "encryptionkey@0"
I20260924 19:31:53.026567 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:53.027288 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:53.027458 5807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:53.027544 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "5a8418dde1c74a8ebfb82f995b77b5f8"
format_stamp: "Formatted at 2026-09-24 19:31:53 on dist-test-slave-m8jb"
server_key: "ebc610c0250b8301183bdf15497bb6a3"
server_key_iv: "0905e0dedda8eca4d390bd1b18fb0f1d"
server_key_version: "encryptionkey@0"
I20260924 19:31:53.027601 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:53.039966 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:53.040262 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:53.040364 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:53.043933 5807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40203
I20260924 19:31:53.043999 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40203 every 8 connection(s)
I20260924 19:31:53.044293 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:53.045436 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:53.047670 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8: Bootstrap starting.
I20260924 19:31:53.047825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 5807
I20260924 19:31:53.047926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:53.048117 392 external_mini_cluster.cc:1428] Setting key c1ec3aea0f21a92b3211f53f63519c89
I20260924 19:31:53.048360 5874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:53.048674 5874 log.cc:824] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:53.049362 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8: No bootstrap required, opened a new log
I20260924 19:31:53.051043 5874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8418dde1c74a8ebfb82f995b77b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40203 } }
I20260924 19:31:53.051172 5874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:53.051203 5874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8418dde1c74a8ebfb82f995b77b5f8, State: Initialized, Role: FOLLOWER
I20260924 19:31:53.051327 5874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8418dde1c74a8ebfb82f995b77b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40203 } }
I20260924 19:31:53.051401 5874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:53.051432 5874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:53.051477 5874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:53.051972 5874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8418dde1c74a8ebfb82f995b77b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40203 } }
I20260924 19:31:53.052062 5874 leader_election.cc:302] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a8418dde1c74a8ebfb82f995b77b5f8; no voters:
I20260924 19:31:53.052206 5874 leader_election.cc:288] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:53.052242 5879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:53.052356 5879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [term 1 LEADER]: Becoming Leader. State: Replica: 5a8418dde1c74a8ebfb82f995b77b5f8, State: Running, Role: LEADER
I20260924 19:31:53.052460 5874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:53.052488 5879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8418dde1c74a8ebfb82f995b77b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40203 } }
I20260924 19:31:53.053241 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:53.048594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.0.98.62:40203)
0924 19:31:53.048821 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:53.048830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:53.048862 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:31:53.049285 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:53.049291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:53.049348 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:53.049475 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:53.049713 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:53.050555 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:53.051313 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:53.051792 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:53.051902 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:53.052579 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:53.052582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:53.052583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:31:53.052788 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:53.052822 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:31:53.052832 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:53.052891 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:53.052963 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 19:31:53.053040 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":118,"threads_started":1}
I20260924 19:31:53.053215 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a8418dde1c74a8ebfb82f995b77b5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8418dde1c74a8ebfb82f995b77b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40203 } } }
I20260924 19:31:53.053287 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:53.053467 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a8418dde1c74a8ebfb82f995b77b5f8. Latest consensus state: current_term: 1 leader_uuid: "5a8418dde1c74a8ebfb82f995b77b5f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8418dde1c74a8ebfb82f995b77b5f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40203 } } }
I20260924 19:31:53.053543 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:53.054208 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 19:31:53.054739 5895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:31:53.054780 5895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:31:53.054831 5885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:53.054996 5885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:53.056275 5885 catalog_manager.cc:1347] Generated new cluster ID: 808eecf6c6ba4551bf536351617d305e
I20260924 19:31:53.056324 5885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:31:53.071087 5885 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:53.071616 5885 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:53.077528 5885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a8418dde1c74a8ebfb82f995b77b5f8: Generated new TSK 0
I20260924 19:31:53.077812 5885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:31:53.128501 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:53.083965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.0.98.62:40203)
0924 19:31:53.084008 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:53.084010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:53.084094 (+ 84us) server_negotiation.cc:407] Connection header received
0924 19:31:53.084157 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:53.084157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:53.084167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:53.084226 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:31:53.084404 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:53.085106 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:53.086261 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:53.086458 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:53.086521 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:53.086688 (+ 167us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:31:53.128203 (+ 41515us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:31:53.128420 (+ 217us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34270: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 19:31:53.128582 5877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34270: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 19:31:53.128584 5903 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:31:53.083869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40203 (local address 127.0.0.1:34270)
0924 19:31:53.084022 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:53.084034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:53.084101 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:53.084243 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:53.084244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:53.084246 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:31:53.084306 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:53.084316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:53.085186 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:53.085187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:53.086157 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:53.086160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:53.086531 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:53.086538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:53.086597 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:53.086603 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:31:53.128371 (+ 41768us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:31:53.128488 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260924 19:31:53.129004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 5807
2026-09-24T19:31:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (218 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (84 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:53 dist-test-slave-m8jb krb5kdc[5915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:53 dist-test-slave-m8jb krb5kdc[5915](info): set up 2 sockets
Sep 24 19:31:53 dist-test-slave-m8jb krb5kdc[5915](info): commencing operation
krb5kdc: starting...
W20260924 19:31:54.256821 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 19:31:54.455793 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.119s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 19:31:54.605065 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.149s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/test-user.keytab.
W20260924 19:31:54.759963 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.155s user 0.000s sys 0.001s
Sep 24 19:31:54 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:31:54.800266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44797
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44797
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:54.877974 5928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:54.878177 5928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:54.878206 5928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:54.879364 5928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:54.879407 5928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:54.879424 5928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:54.879439 5928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:54.879455 5928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:54.880985 5928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:54.881220 5928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:54.881420 5928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:31:54.886080 5928 server_base.cc:1034] running on GCE node
W20260924 19:31:54.886049 5933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:54.886039 5936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:54.886039 5934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:54.886490 5928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:54.886754 5928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:54.887912 5928 hybrid_clock.cc:648] HybridClock initialized: now 1790278314887897 us; error 27 us; skew 500 ppm
Sep 24 19:31:54 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:54.890379 5928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:31:54.890791 5928 webserver.cc:460] Webserver started at http://127.0.98.62:44803/ using document root <none> and password file <none>
I20260924 19:31:54.891016 5928 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:54.891062 5928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:54.891179 5928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:54.892213 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce"
format_stamp: "Formatted at 2026-09-24 19:31:54 on dist-test-slave-m8jb"
server_key: "cce806d95100460d6ca2276ce5befcea"
server_key_iv: "e3e83ffb6ae5e5f10321e886a5bdbe19"
server_key_version: "encryptionkey@0"
I20260924 19:31:54.892545 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce"
format_stamp: "Formatted at 2026-09-24 19:31:54 on dist-test-slave-m8jb"
server_key: "cce806d95100460d6ca2276ce5befcea"
server_key_iv: "e3e83ffb6ae5e5f10321e886a5bdbe19"
server_key_version: "encryptionkey@0"
I20260924 19:31:54.893777 5928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:31:54.894441 5943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:54.894586 5928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:54.894658 5928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce"
format_stamp: "Formatted at 2026-09-24 19:31:54 on dist-test-slave-m8jb"
server_key: "cce806d95100460d6ca2276ce5befcea"
server_key_iv: "e3e83ffb6ae5e5f10321e886a5bdbe19"
server_key_version: "encryptionkey@0"
I20260924 19:31:54.894721 5928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:54.910849 5928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:54.912459 5928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:54.912595 5928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:54.916270 5928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44797
I20260924 19:31:54.916320 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44797 every 8 connection(s)
I20260924 19:31:54.916803 5928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:54.917413 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:54.919580 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce: Bootstrap starting.
I20260924 19:31:54.920128 5996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:54.920383 5996 log.cc:824] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:54.921027 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce: No bootstrap required, opened a new log
I20260924 19:31:54.922530 5996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44797 } }
I20260924 19:31:54.922663 5996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:54.922695 5996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c0b2c62d9e4a91b68a9b2b61f77fce, State: Initialized, Role: FOLLOWER
I20260924 19:31:54.922825 5996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44797 } }
I20260924 19:31:54.922901 5996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:54.922923 5996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:54.922968 5996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:54.923777 5996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44797 } }
I20260924 19:31:54.923873 5996 leader_election.cc:302] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36c0b2c62d9e4a91b68a9b2b61f77fce; no voters:
I20260924 19:31:54.924022 5996 leader_election.cc:288] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:54.924050 5999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:54.924185 5999 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [term 1 LEADER]: Becoming Leader. State: Replica: 36c0b2c62d9e4a91b68a9b2b61f77fce, State: Running, Role: LEADER
I20260924 19:31:54.924240 5996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:54.924487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 5928
I20260924 19:31:54.924588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:54.924806 392 external_mini_cluster.cc:1428] Setting key e6c22cf37b2a6c2746880d46cf94d6c0
I20260924 19:31:54.924268 5999 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44797 } }
I20260924 19:31:54.925503 6001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c0b2c62d9e4a91b68a9b2b61f77fce. Latest consensus state: current_term: 1 leader_uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44797 } } }
I20260924 19:31:54.925578 6000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c0b2c62d9e4a91b68a9b2b61f77fce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44797 } } }
I20260924 19:31:54.925702 6001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:54.925709 6000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce [sys.catalog]: This master's current role is: LEADER
W20260924 19:31:54.925941 6014 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:31:54.925987 6014 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:31:54.928758 6017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:54.928996 6017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:54.931028 6017 catalog_manager.cc:1347] Generated new cluster ID: 96582ee6f5c748eca3378ffb7956c346
I20260924 19:31:54.931071 6017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:31:54 dist-test-slave-m8jb krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:54.935465 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:54.925407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.0.98.62:44797)
0924 19:31:54.925778 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:54.925788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:54.925817 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:31:54.925933 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:54.925940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:54.925996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:54.926137 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:54.926827 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:54.927737 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:54.928816 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:54.929413 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:54.929516 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:54.932133 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:54.932137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:54.932137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:54.932148 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:54.933941 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:54.934314 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:54.934317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:54.934317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:54.934418 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:54.934608 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:54.934609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:54.934609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:54.934873 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:54.935010 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:54.935112 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 19:31:54.935234 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":64,"threads_started":1}
I20260924 19:31:54.956581 6017 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:54.957332 6017 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:54.962929 6017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36c0b2c62d9e4a91b68a9b2b61f77fce: Generated new TSK 0
I20260924 19:31:54.963111 6017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:31:54.991504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44797
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:55.072877 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:55.073150 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:55.073185 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:55.074734 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:55.074779 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:55.074805 6022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:55.076510 6022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:55.076802 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:55.077033 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:55.081977 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:55.082101 6022 server_base.cc:1034] running on GCE node
W20260924 19:31:55.082849 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:55.082964 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:55.083196 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:55.083472 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:55.084650 6022 hybrid_clock.cc:648] HybridClock initialized: now 1790278315084614 us; error 31 us; skew 500 ppm
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.086997 6022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:31:55.087435 6022 webserver.cc:460] Webserver started at http://127.0.98.1:39127/ using document root <none> and password file <none>
I20260924 19:31:55.087626 6022 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:55.087672 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:55.087815 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:55.088949 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "a7fc98fdd32747cbaa54b7d2152dc0fd"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "1791f1c33627fd33b7dac24977d980db"
server_key_iv: "08fff58dfd7392173e279fa7d8c9bb75"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.089273 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a7fc98fdd32747cbaa54b7d2152dc0fd"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "1791f1c33627fd33b7dac24977d980db"
server_key_iv: "08fff58dfd7392173e279fa7d8c9bb75"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.090572 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:55.091254 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.091414 6022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:55.091495 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "a7fc98fdd32747cbaa54b7d2152dc0fd"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "1791f1c33627fd33b7dac24977d980db"
server_key_iv: "08fff58dfd7392173e279fa7d8c9bb75"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.091557 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:55.101958 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:55.103724 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:55.103861 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:55.104067 6022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:55.104346 6022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:55.104377 6022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.104405 6022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:55.104432 6022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.109921 6022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36705
I20260924 19:31:55.109977 6150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36705 every 8 connection(s)
I20260924 19:31:55.110260 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:31:55.115856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 6022
I20260924 19:31:55.115939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:55.116112 392 external_mini_cluster.cc:1428] Setting key 3dbbdbe91c0dd7199df0e8635df3aaf1
I20260924 19:31:55.119100 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.110916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47041 (local address 127.0.98.62:44797)
0924 19:31:55.110997 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.110999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.111174 (+ 175us) server_negotiation.cc:407] Connection header received
0924 19:31:55.112072 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.112073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.112084 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.112132 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.112744 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.113248 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.114273 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.114485 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.114548 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.116525 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:55.116527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:55.116527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:55.116535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:55.117844 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.118214 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.118215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.118215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.118266 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.118494 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.118495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.118495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.118599 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:55.118654 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:55.118922 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 19:31:55.119011 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:31:55.119254 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.110900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44797 (local address 127.0.98.1:47041)
0924 19:31:55.111058 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:55.111094 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:55.111975 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:55.112195 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:55.112198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:55.112437 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:55.112620 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:55.112624 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.113329 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.113330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:55.114180 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:55.114182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.114565 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.114566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:55.114666 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.115135 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:55.115142 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:55.116432 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:55.117924 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:55.117931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:55.117933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:55.118143 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:55.118321 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:55.118322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:55.118322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:55.118433 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:55.118654 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:55.118661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:55.118793 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:31:55.118889 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:31:55.119041 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260924 19:31:55.119719 6151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44797
I20260924 19:31:55.119802 6151 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:55.119969 6151 heartbeater.cc:511] Master 127.0.98.62:44797 requested a full tablet report, sending...
I20260924 19:31:55.120527 5960 ts_manager.cc:194] Registered new tserver with Master: a7fc98fdd32747cbaa54b7d2152dc0fd (127.0.98.1:36705)
I20260924 19:31:55.121438 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47041
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:31:55.139663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44797
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:55.217620 6156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:55.217841 6156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:55.217865 6156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:55.219000 6156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:55.219036 6156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:55.219060 6156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:55.220455 6156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:55.220654 6156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:55.220840 6156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:55.225514 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:55.225538 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:55.225726 6156 server_base.cc:1034] running on GCE node
W20260924 19:31:55.225772 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:55.225961 6156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:55.226179 6156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:55.227308 6156 hybrid_clock.cc:648] HybridClock initialized: now 1790278315227292 us; error 25 us; skew 500 ppm
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.229694 6156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:31:55.230082 6156 webserver.cc:460] Webserver started at http://127.0.98.2:40847/ using document root <none> and password file <none>
I20260924 19:31:55.230270 6156 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:55.230316 6156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:55.230445 6156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:55.231472 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "10168601197640eeab8c5666a70f1486"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "ec1c691f12dd417a762b837a4bd61509"
server_key_iv: "247efea4506ce868ae76c5f7ba3b2d97"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.231791 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "10168601197640eeab8c5666a70f1486"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "ec1c691f12dd417a762b837a4bd61509"
server_key_iv: "247efea4506ce868ae76c5f7ba3b2d97"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.233048 6156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:55.233671 6171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.233827 6156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:55.233907 6156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "10168601197640eeab8c5666a70f1486"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "ec1c691f12dd417a762b837a4bd61509"
server_key_iv: "247efea4506ce868ae76c5f7ba3b2d97"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.233968 6156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:55.269026 6156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:55.271169 6156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:55.271306 6156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:55.271538 6156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:55.271819 6156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:55.271852 6156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.271879 6156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:55.271896 6156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.277369 6156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38075
I20260924 19:31:55.277447 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38075 every 8 connection(s)
I20260924 19:31:55.277735 6156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:55.284394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 6156
I20260924 19:31:55.284492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:55.284709 392 external_mini_cluster.cc:1428] Setting key c636433538f76b505c01a95061fc3f23
I20260924 19:31:55.286973 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.278523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55911 (local address 127.0.98.62:44797)
0924 19:31:55.278576 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.278578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.278775 (+ 197us) server_negotiation.cc:407] Connection header received
0924 19:31:55.279726 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.279728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.279739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.279794 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.280405 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.281016 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.281964 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.282157 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.282212 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.284188 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:55.284189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:55.284190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:55.284197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:55.285730 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.286096 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.286097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.286098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.286155 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.286348 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.286348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.286349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.286472 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:55.286528 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:55.286775 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 19:31:55.286882 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:31:55.287099 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.278407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44797 (local address 127.0.98.2:55911)
0924 19:31:55.278597 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:55.278650 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:55.279592 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:55.279863 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:55.279866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:55.280108 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:55.280322 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:55.280327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.281086 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.281087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:55.281893 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:55.281895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.282220 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.282220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:55.282329 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.282761 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:55.282770 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:55.284096 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:55.285809 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:55.285815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:55.285817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:55.286031 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:55.286207 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:55.286207 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:55.286208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:55.286299 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:55.286525 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:55.286529 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:55.286648 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 19:31:55.286736 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:31:55.286886 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260924 19:31:55.287652 6285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44797
I20260924 19:31:55.287748 6285 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:55.287937 6285 heartbeater.cc:511] Master 127.0.98.62:44797 requested a full tablet report, sending...
I20260924 19:31:55.288311 5960 ts_manager.cc:194] Registered new tserver with Master: 10168601197640eeab8c5666a70f1486 (127.0.98.2:38075)
I20260924 19:31:55.288723 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55911
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:31:55.309262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44797
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:55.387089 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:55.387275 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:55.387296 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:55.388391 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:55.388424 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:55.388448 6290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:55.389926 6290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44797
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:55.390138 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:55.390437 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:31:55.395220 6290 server_base.cc:1034] running on GCE node
W20260924 19:31:55.395329 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:55.395300 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:55.395397 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:55.395618 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:55.395814 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:55.396960 6290 hybrid_clock.cc:648] HybridClock initialized: now 1790278315396963 us; error 43 us; skew 500 ppm
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.399273 6290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:31:55.399657 6290 webserver.cc:460] Webserver started at http://127.0.98.3:36547/ using document root <none> and password file <none>
I20260924 19:31:55.399837 6290 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:55.399883 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:55.399969 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:55.400974 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9bf8d70995c4e18af730947e471fefc"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "e2bd85302445202a39cc5c1e61335ef0"
server_key_iv: "f481134677f79b342746b2de28615d8a"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.401253 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9bf8d70995c4e18af730947e471fefc"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "e2bd85302445202a39cc5c1e61335ef0"
server_key_iv: "f481134677f79b342746b2de28615d8a"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.402371 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:55.403014 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.403174 6290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:55.403249 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "f9bf8d70995c4e18af730947e471fefc"
format_stamp: "Formatted at 2026-09-24 19:31:55 on dist-test-slave-m8jb"
server_key: "e2bd85302445202a39cc5c1e61335ef0"
server_key_iv: "f481134677f79b342746b2de28615d8a"
server_key_version: "encryptionkey@0"
I20260924 19:31:55.403311 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:55.425415 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:55.427532 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:55.427672 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:55.427906 6290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:55.428200 6290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:55.428234 6290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.428264 6290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:55.428282 6290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:55.433827 6290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39929
I20260924 19:31:55.433866 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39929 every 8 connection(s)
I20260924 19:31:55.434192 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:31:55.442854 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.434895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35683 (local address 127.0.98.62:44797)
0924 19:31:55.434948 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.434950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.435131 (+ 181us) server_negotiation.cc:407] Connection header received
0924 19:31:55.436100 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.436102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.436114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.436165 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.436934 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.437444 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.438465 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.438656 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.438705 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.440719 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:55.440720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:55.440720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:55.440726 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:55.441815 (+ 1089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.442154 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.442155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.442155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.442201 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.442380 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.442380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.442380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.442474 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:55.442518 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:55.442731 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 19:31:55.442787 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:31:55.443045 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.434851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44797 (local address 127.0.98.3:35683)
0924 19:31:55.435016 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:55.435053 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:55.435991 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:55.436257 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:55.436261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:55.436597 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:55.436839 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:55.436844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.437545 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.437546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:55.438379 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:55.438397 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.438716 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.438717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:55.438814 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.439310 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:55.439318 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:55.440588 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:55.441885 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:55.441890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:55.441892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:55.442099 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:55.442247 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:55.442247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:55.442247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:55.442335 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:55.442517 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:55.442520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:55.442634 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 19:31:55.442711 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:31:55.442829 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260924 19:31:55.443499 6419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44797
I20260924 19:31:55.443593 6419 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:55.443760 6419 heartbeater.cc:511] Master 127.0.98.62:44797 requested a full tablet report, sending...
I20260924 19:31:55.443755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 6290
I20260924 19:31:55.443817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:55.444011 392 external_mini_cluster.cc:1428] Setting key c897af1a0e6f0a0013e676344b1974da
I20260924 19:31:55.444191 5960 ts_manager.cc:194] Registered new tserver with Master: f9bf8d70995c4e18af730947e471fefc (127.0.98.3:39929)
I20260924 19:31:55.444648 5960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35683
I20260924 19:31:55.444845 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/test-admin.keytab.
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.459053 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 19:31:55.460537 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:31:55.459870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.0.98.1:36705)
0924 19:31:55.460041 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.460045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.460054 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:55.460082 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.460083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.460097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.460159 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.460454 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34940: BlockingRecv error: recv got EOF from 127.0.0.1:34940 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
W20260924 19:31:55.460726 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.461635 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.462512 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:55.469623 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.463151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34950 (local address 127.0.98.1:36705)
0924 19:31:55.463197 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.463199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.463209 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:31:55.463253 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.463253 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.463264 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.463320 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.463723 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.464442 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.465176 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.465494 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.465546 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.467337 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:55.467344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:55.467344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:55.467350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:55.468687 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.468990 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.468993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.468993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.469059 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.469195 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.469195 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.469196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.469324 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:55.469451 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:55.469489 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 19:31:55.469550 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260924 19:31:55.471179 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:31:55.470742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.0.98.1:36705)
0924 19:31:55.470774 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.470776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.470786 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:31:55.470837 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.470837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.470846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.470885 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.471133 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34960: BlockingRecv error: recv got EOF from 127.0.0.1:34960 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 19:31:55.471251 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.472112 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.472916 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:55.479007 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.473386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.0.98.1:36705)
0924 19:31:55.473436 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.473438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.473445 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:31:55.473498 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.473499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.473507 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.473549 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.473826 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.474297 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.474927 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.475103 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.475143 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.476977 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:55.476978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:55.476979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:55.476984 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:55.478101 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.478379 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.478380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.478380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.478433 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.478565 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.478566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.478566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.478645 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:55.478705 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:55.478888 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:31:55.478946 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 19:31:55.480396 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:31:55.479951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.0.98.1:36705)
0924 19:31:55.479993 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.479994 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.480030 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:31:55.480080 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.480081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.480088 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.480148 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.480347 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34988: BlockingRecv error: recv got EOF from 127.0.0.1:34988 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 19:31:55.480571 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790278290121024-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.481470 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:55.482225 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 19:31:55.488243 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:55.482733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34996 (local address 127.0.98.1:36705)
0924 19:31:55.482767 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:55.482769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:55.482793 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:31:55.482839 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:55.482840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:55.482847 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:55.482898 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:55.483184 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.483688 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.484323 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:55.484509 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:55.484556 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:55.486316 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:55.486317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:55.486317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:55.486323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:55.487489 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.487733 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.487734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.487734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.487778 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:55.487915 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:55.487916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:55.487916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:55.487990 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:55.488049 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:55.488135 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 19:31:55.488184 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:55.488875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 6022
I20260924 19:31:55.493968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 6156
I20260924 19:31:55.497689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 6290
I20260924 19:31:55.501528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 5928
2026-09-24T19:31:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2201 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[6439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[6439](info): set up 2 sockets
Sep 24 19:31:55 dist-test-slave-m8jb krb5kdc[6439](info): commencing operation
krb5kdc: starting...
W20260924 19:31:56.444582 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:31:56 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:31:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 19:31:56.555922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:56.641168 6452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:56.641360 6452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:56.641381 6452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:56.642494 6452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:56.642530 6452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:56.642544 6452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:56.642565 6452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:56.642589 6452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:56.644044 6452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:56.644266 6452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:56.644482 6452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:31:56.648977 6452 server_base.cc:1034] running on GCE node
W20260924 19:31:56.649101 6460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:56.648969 6458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:56.649075 6457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:56.649389 6452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:56.649649 6452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:56.650815 6452 hybrid_clock.cc:648] HybridClock initialized: now 1790278316650791 us; error 35 us; skew 500 ppm
Sep 24 19:31:56 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:56.653169 6452 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 19:31:56.653563 6452 webserver.cc:460] Webserver started at http://127.0.98.62:37695/ using document root <none> and password file <none>
I20260924 19:31:56.653815 6452 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:56.653867 6452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:56.653986 6452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:56.655001 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "8b5cdc2713da469c9dcb55b7726d86ba"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "b09ad000216a39b9871226b66c82c90f"
server_key_iv: "a007bab7d84f1c605dbe414e055fa781"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.655318 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b5cdc2713da469c9dcb55b7726d86ba"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "b09ad000216a39b9871226b66c82c90f"
server_key_iv: "a007bab7d84f1c605dbe414e055fa781"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.656526 6452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:31:56.657253 6467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:56.657406 6452 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:56.657488 6452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "8b5cdc2713da469c9dcb55b7726d86ba"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "b09ad000216a39b9871226b66c82c90f"
server_key_iv: "a007bab7d84f1c605dbe414e055fa781"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.657545 6452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:56.669106 6452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:56.670938 6452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:56.671066 6452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:56.674644 6452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43727
I20260924 19:31:56.674677 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43727 every 8 connection(s)
I20260924 19:31:56.674999 6452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:31:56.675927 6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:56.678215 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: Bootstrap starting.
I20260924 19:31:56.678769 6520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:56.679035 6520 log.cc:824] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:56.679661 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: No bootstrap required, opened a new log
I20260924 19:31:56.680336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 6452
I20260924 19:31:56.680438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:31:56.680610 392 external_mini_cluster.cc:1428] Setting key 9ab0fa2a0b401393ad380c9c46a8e325
I20260924 19:31:56.681528 6520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } }
I20260924 19:31:56.681695 6520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:56.681730 6520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b5cdc2713da469c9dcb55b7726d86ba, State: Initialized, Role: FOLLOWER
I20260924 19:31:56.681850 6520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } }
I20260924 19:31:56.681934 6520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:31:56.681957 6520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:31:56.682000 6520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:56.682991 6520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } }
I20260924 19:31:56.683094 6520 leader_election.cc:302] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b5cdc2713da469c9dcb55b7726d86ba; no voters:
I20260924 19:31:56.683250 6520 leader_election.cc:288] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:31:56.683308 6525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:56.683440 6525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 1 LEADER]: Becoming Leader. State: Replica: 8b5cdc2713da469c9dcb55b7726d86ba, State: Running, Role: LEADER
I20260924 19:31:56.683470 6520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:56.683532 6525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } }
I20260924 19:31:56.683858 6525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b5cdc2713da469c9dcb55b7726d86ba. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } }
I20260924 19:31:56.683924 6525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:56.684103 6532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:31:56.684782 6532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:31:56.684870 6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } }
I20260924 19:31:56.685217 6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:56.686923 6532 catalog_manager.cc:1347] Generated new cluster ID: af5bc9eb2aad45d0a00044e1d10d36dc
I20260924 19:31:56.686964 6532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:31:56 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:31:56.691476 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:56.681145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.0.98.62:43727)
0924 19:31:56.681345 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:56.681355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:56.681400 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:31:56.681514 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:56.681520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:56.681576 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:56.681725 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:56.682443 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.683771 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.684775 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.685392 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.685507 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:56.688041 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:56.688045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:56.688045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:56.688057 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:56.689942 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:56.690324 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:56.690326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:56.690327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:56.690445 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:56.690646 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:56.690647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:56.690647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:56.690922 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:56.691056 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:56.691162 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 19:31:56.691289 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260924 19:31:56.693056 6532 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:31:56.693533 6532 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:31:56.705788 6532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: Generated new TSK 0
I20260924 19:31:56.705953 6532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 19:31:56.731153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43727
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:56.810655 6546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:56.810866 6546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:56.810890 6546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:56.811986 6546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:56.812021 6546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:56.812044 6546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:56.813484 6546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:56.813679 6546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:56.813863 6546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:56.818704 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:56.818683 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:56.818722 6546 server_base.cc:1034] running on GCE node
W20260924 19:31:56.818688 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:56.819085 6546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:56.819346 6546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:56.820493 6546 hybrid_clock.cc:648] HybridClock initialized: now 1790278316820482 us; error 30 us; skew 500 ppm
Sep 24 19:31:56 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:56.822909 6546 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 19:31:56.823335 6546 webserver.cc:460] Webserver started at http://127.0.98.1:43673/ using document root <none> and password file <none>
I20260924 19:31:56.823545 6546 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:56.823593 6546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:56.823717 6546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:56.824827 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "30a1e8e885b14651aac5c8f8d58ac786"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "7f69674d2d4da3561cbe40d5cc717c1c"
server_key_iv: "e42393148b9dfe6b044c4dc90b35c08a"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.825171 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "30a1e8e885b14651aac5c8f8d58ac786"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "7f69674d2d4da3561cbe40d5cc717c1c"
server_key_iv: "e42393148b9dfe6b044c4dc90b35c08a"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.826444 6546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:56.827163 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:56.827337 6546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:56.827416 6546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "30a1e8e885b14651aac5c8f8d58ac786"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "7f69674d2d4da3561cbe40d5cc717c1c"
server_key_iv: "e42393148b9dfe6b044c4dc90b35c08a"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.827477 6546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:56.840544 6546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:56.842504 6546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:56.842648 6546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:56.842870 6546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:56.843168 6546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:56.843200 6546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:56.843230 6546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:56.843248 6546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:56.848671 6546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38451
I20260924 19:31:56.848773 6674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38451 every 8 connection(s)
I20260924 19:31:56.849050 6546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:31:56 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:31:56.855687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 6546
I20260924 19:31:56.855791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:31:56.856000 392 external_mini_cluster.cc:1428] Setting key 55434d670767897c36946affe65b5636
I20260924 19:31:56.857824 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:56.849748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40059 (local address 127.0.98.62:43727)
0924 19:31:56.849792 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:56.849794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:56.849968 (+ 174us) server_negotiation.cc:407] Connection header received
0924 19:31:56.850445 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:56.850447 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:56.850456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:56.850505 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:56.851381 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.851867 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.852927 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.853122 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.853172 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:56.855083 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:56.855085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:56.855085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:56.855093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:56.856442 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:56.856910 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:56.856911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:56.856912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:56.856981 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:56.857250 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:56.857250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:56.857251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:56.857350 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:56.857398 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:56.857691 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 19:31:56.857760 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:56.858211 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:56.849721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:40059)
0924 19:31:56.849866 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:56.849902 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:56.850369 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:56.850562 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:56.850565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:56.851112 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:56.851315 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:56.851320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.851934 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.851935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:56.852861 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:56.852864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.853183 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.853184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:56.853284 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:56.853720 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:56.853727 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:56.854996 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:56.856508 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:56.856515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:56.856517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:56.856811 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:56.857047 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:56.857048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:56.857049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:56.857195 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:56.857401 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:56.857406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:56.857568 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 19:31:56.857676 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:31:56.857837 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260924 19:31:56.858654 6675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43727
I20260924 19:31:56.858748 6675 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:56.858927 6675 heartbeater.cc:511] Master 127.0.98.62:43727 requested a full tablet report, sending...
I20260924 19:31:56.859421 6484 ts_manager.cc:194] Registered new tserver with Master: 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451)
I20260924 19:31:56.860116 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40059
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 19:31:56.879741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43727
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:56.957896 6680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:56.958103 6680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:56.958133 6680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:56.959276 6680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:56.959318 6680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:56.959352 6680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:56.960836 6680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:56.961066 6680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:56.961274 6680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:56.966150 6688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:56.966215 6685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:56.966166 6686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:56.966244 6680 server_base.cc:1034] running on GCE node
I20260924 19:31:56.966534 6680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:56.966741 6680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:56.967896 6680 hybrid_clock.cc:648] HybridClock initialized: now 1790278316967881 us; error 29 us; skew 500 ppm
Sep 24 19:31:56 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:56.970230 6680 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 19:31:56.970636 6680 webserver.cc:460] Webserver started at http://127.0.98.2:44753/ using document root <none> and password file <none>
I20260924 19:31:56.970844 6680 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:56.970902 6680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:56.971024 6680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:56.972054 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "a77223cab72e427ba7553a4480fe9d93"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "f7f75c39e802ac70548b7a1af15aec4b"
server_key_iv: "4b35cf5cd93bed3fd3ded23ee38d2615"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.972388 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a77223cab72e427ba7553a4480fe9d93"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "f7f75c39e802ac70548b7a1af15aec4b"
server_key_iv: "4b35cf5cd93bed3fd3ded23ee38d2615"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.973699 6680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:56.974403 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:56.974539 6680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:56.974619 6680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "a77223cab72e427ba7553a4480fe9d93"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "f7f75c39e802ac70548b7a1af15aec4b"
server_key_iv: "4b35cf5cd93bed3fd3ded23ee38d2615"
server_key_version: "encryptionkey@0"
I20260924 19:31:56.974685 6680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:56.987150 6680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:56.988897 6680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:56.989030 6680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:56.989251 6680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:56.989565 6680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:56.989598 6680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:56.989627 6680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:56.989643 6680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:56.995000 6680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39351
I20260924 19:31:56.995059 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39351 every 8 connection(s)
I20260924 19:31:56.995357 6680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:31:56 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:31:57.003989 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:56.996060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45865 (local address 127.0.98.62:43727)
0924 19:31:56.996121 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:56.996123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:56.996354 (+ 231us) server_negotiation.cc:407] Connection header received
0924 19:31:56.996847 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:56.996848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:56.996857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:56.996917 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:56.997795 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.998295 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.999282 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.999480 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.999530 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.001634 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.001635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.001636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.001642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.002747 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.003119 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.003121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.003121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.003172 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.003386 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.003387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.003388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.003494 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.003548 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.003806 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.003899 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:31:57.004148 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:56.996038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:45865)
0924 19:31:56.996202 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:56.996245 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:56.996762 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:56.996974 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:56.996977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:56.997527 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:56.997727 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:56.997732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.998376 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.998377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:56.999207 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:56.999209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:56.999575 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:56.999577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:56.999694 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.000191 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.000199 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.001541 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.002827 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.002834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.002836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.003047 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.003228 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.003229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.003230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.003328 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.003551 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.003557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.003691 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.003776 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.003929 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260924 19:31:57.004364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 6680
I20260924 19:31:57.004482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:31:57.004660 6809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43727
I20260924 19:31:57.004761 6809 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:57.004768 392 external_mini_cluster.cc:1428] Setting key dddd7613c228865a7ea15030db70c661
I20260924 19:31:57.004974 6809 heartbeater.cc:511] Master 127.0.98.62:43727 requested a full tablet report, sending...
I20260924 19:31:57.005328 6484 ts_manager.cc:194] Registered new tserver with Master: a77223cab72e427ba7553a4480fe9d93 (127.0.98.2:39351)
I20260924 19:31:57.005676 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45865
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 19:31:57.029095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43727
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:57.109279 6814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:57.109490 6814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:57.109512 6814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:57.110596 6814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:57.110628 6814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:57.110651 6814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:57.112038 6814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:57.112221 6814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:57.112401 6814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:57.117203 6820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.117255 6814 server_base.cc:1034] running on GCE node
W20260924 19:31:57.117192 6822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.117169 6819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.117554 6814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:57.117779 6814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:57.118948 6814 hybrid_clock.cc:648] HybridClock initialized: now 1790278317118939 us; error 30 us; skew 500 ppm
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:57.121474 6814 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 19:31:57.121899 6814 webserver.cc:460] Webserver started at http://127.0.98.3:36059/ using document root <none> and password file <none>
I20260924 19:31:57.122097 6814 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.122144 6814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.122279 6814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:57.123303 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d22290598ca4676a43fd50dc9cfc77e"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "670df4ccfed6bd2807c4325c1d93136a"
server_key_iv: "fe0dc3613460a14def9f342789d90092"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.123633 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d22290598ca4676a43fd50dc9cfc77e"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "670df4ccfed6bd2807c4325c1d93136a"
server_key_iv: "fe0dc3613460a14def9f342789d90092"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.124910 6814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:31:57.125576 6829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.125725 6814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:57.125800 6814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "1d22290598ca4676a43fd50dc9cfc77e"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "670df4ccfed6bd2807c4325c1d93136a"
server_key_iv: "fe0dc3613460a14def9f342789d90092"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.125862 6814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.137658 6814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:57.139710 6814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:57.139847 6814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:57.140046 6814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:57.140334 6814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:57.140365 6814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.140394 6814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:57.140411 6814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.145812 6814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41535
I20260924 19:31:57.145905 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41535 every 8 connection(s)
I20260924 19:31:57.146168 6814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:31:57.153843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 6814
I20260924 19:31:57.153949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:31:57.154121 392 external_mini_cluster.cc:1428] Setting key 4d27dee6d4fc97022dee187637b93940
I20260924 19:31:57.154466 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.146825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52613 (local address 127.0.98.62:43727)
0924 19:31:57.146867 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.146869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.147057 (+ 188us) server_negotiation.cc:407] Connection header received
0924 19:31:57.147516 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.147517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.147526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.147568 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.148446 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.148986 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.149912 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.150087 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.150130 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.152002 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.152004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.152004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.152010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.153263 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.153589 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.153590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.153590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.153640 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.153832 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.153833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.153833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.153965 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.154028 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.154298 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.154374 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:31:57.154632 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.146794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:52613)
0924 19:31:57.146936 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.146970 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.147439 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.147631 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.147635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.148181 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.148371 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.148384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.149058 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.149058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.149851 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.149853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.150139 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.150140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.150231 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.150649 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.150657 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.151922 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.153326 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.153332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.153333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.153533 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.153694 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.153695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.153695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.153780 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.154019 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.154025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.154151 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.154256 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.154412 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260924 19:31:57.155020 6943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43727
I20260924 19:31:57.155098 6943 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:57.155252 6943 heartbeater.cc:511] Master 127.0.98.62:43727 requested a full tablet report, sending...
I20260924 19:31:57.155588 6483 ts_manager.cc:194] Registered new tserver with Master: 1d22290598ca4676a43fd50dc9cfc77e (127.0.98.3:41535)
I20260924 19:31:57.155944 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52613
I20260924 19:31:57.165628 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 19:31:57.190860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45165
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43727,127.0.98.61:45165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:57.269037 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:57.269243 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:57.269269 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:57.270371 6948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:57.270414 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:57.270432 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:57.270447 6948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:57.270462 6948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:57.271854 6948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43727,127.0.98.61:45165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
W20260924 19:31:57.271974 6948 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43727,127.0.98.61:45165'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 19:31:57.272115 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:57.272303 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:57.276813 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.276829 6956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.276872 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.277021 6948 server_base.cc:1034] running on GCE node
I20260924 19:31:57.277252 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:57.277509 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:57.278664 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790278317278664 us; error 40 us; skew 500 ppm
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:57.281090 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 19:31:57.281524 6948 webserver.cc:460] Webserver started at http://127.0.98.61:39783/ using document root <none> and password file <none>
I20260924 19:31:57.281723 6948 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.281769 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.281903 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:57.282975 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data/instance:
uuid: "6fb75f62318049e8b2d64c65559219d2"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1ddfdb7ed19562a285a8496e1722417"
server_key_iv: "30f47362536432e7930797d7600dc477"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.283315 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal/instance:
uuid: "6fb75f62318049e8b2d64c65559219d2"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1ddfdb7ed19562a285a8496e1722417"
server_key_iv: "30f47362536432e7930797d7600dc477"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.284729 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:31:57.285581 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.285796 6948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:57.285878 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
uuid: "6fb75f62318049e8b2d64c65559219d2"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1ddfdb7ed19562a285a8496e1722417"
server_key_iv: "30f47362536432e7930797d7600dc477"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.285956 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.300197 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:57.302382 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:57.302501 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:57.306439 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45165
I20260924 19:31:57.306470 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45165 every 8 connection(s)
I20260924 19:31:57.306795 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:31:57.307717 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:57.309290 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:31:57.315689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 6948
I20260924 19:31:57.318061 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.310024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.0.98.62:43727)
0924 19:31:57.310083 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.310085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.310261 (+ 176us) server_negotiation.cc:407] Connection header received
0924 19:31:57.310708 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.310710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.310720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.310780 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.311565 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.312170 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.313192 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.313382 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.313434 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.315445 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.315446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.315447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.315454 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.316852 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.317196 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.317197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.317198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.317248 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.317460 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.317460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.317461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.317593 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.317650 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.317902 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.317975 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:31:57.318264 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.309985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:40764)
0924 19:31:57.310136 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.310177 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.310618 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.310844 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.310847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.311184 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.311468 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.311475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.312245 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.312246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.313123 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.313125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.313443 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.313443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.313538 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.314013 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.314021 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.315365 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.316920 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.316926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.316928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.317129 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.317309 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.317310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.317310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.317401 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.317654 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.317660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.317782 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.317868 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.318012 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260924 19:31:57.318864 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 19:31:57.323915 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.316125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.0.98.61:45165)
0924 19:31:57.316285 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.316288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.316302 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:31:57.316318 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.316320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.316329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.316396 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.316973 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.317624 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.318428 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.318771 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.318831 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.321067 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.321074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.321074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.321081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.322710 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.323003 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.323006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.323006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.323093 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.323319 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.323320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.323321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.323503 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.323682 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.323727 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.323809 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260924 19:31:57.325737 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.319089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.0.1:56362)
0924 19:31:57.319133 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.319142 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.319179 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.319514 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.319515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.319640 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.319683 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.319684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.320342 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.320343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.320932 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.320934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.321261 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.321262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.321321 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.321644 (+ 323us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.321647 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.322923 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.325040 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.325041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.325041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.325185 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.325332 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.325332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.325332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.325373 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.325539 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.325540 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.325570 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.325612 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.325680 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:31:57.325866 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.319124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.0.98.61:45165)
0924 19:31:57.319418 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.319420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.319425 (+ 5us) server_negotiation.cc:407] Connection header received
0924 19:31:57.319438 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.319439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.319446 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.319500 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.319742 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.320249 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.321004 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.321197 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.321255 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.322999 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.323000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.323000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.323006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.324979 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.325243 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.325243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.325244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.325288 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.325421 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.325421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.325422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.325500 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.325540 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.325629 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.325758 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":276,"thread_start_us":95,"threads_started":1}
I20260924 19:31:57.326992 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Bootstrap starting.
I20260924 19:31:57.327555 7016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:57.327778 7016 log.cc:824] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:57.328439 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: No bootstrap required, opened a new log
I20260924 19:31:57.329926 7016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } }
I20260924 19:31:57.330040 7016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:57.330061 7016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb75f62318049e8b2d64c65559219d2, State: Initialized, Role: FOLLOWER
I20260924 19:31:57.330147 7016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } }
I20260924 19:31:57.330404 7016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:57.330361 7023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } } }
I20260924 19:31:57.330504 7023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:31:57.332104 7034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:31:57.332156 7034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:31:57.333472 6948 master_runner.cc:428] Detected that this master 6fb75f62318049e8b2d64c65559219d2 is joining an existing cluster
I20260924 19:31:57.333521 6948 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45165
I20260924 19:31:57.341135 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.334535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:40780)
0924 19:31:57.334721 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.334729 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.334796 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.334955 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.334955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.335195 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.335251 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.335252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.336179 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.336180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.337122 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.337125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.337466 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.337467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.337544 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.337950 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.337955 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.338322 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.339943 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.339943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.339944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.340153 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.340358 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.340359 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.340359 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.340416 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.340729 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.340730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.340789 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.340923 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.341015 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260924 19:31:57.341157 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.334632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.0.98.62:43727)
0924 19:31:57.334677 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.334679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.334774 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:31:57.334884 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.334886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.334897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.334940 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.335319 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.336092 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.337201 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.337396 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.337455 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.338432 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.338434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.338435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.338442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.339853 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.340235 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.340236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.340236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.340291 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.340487 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.340488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.340488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.340615 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.340703 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.340984 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.341069 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:31:57.342352 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.334673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.0.98.61:45165)
0924 19:31:57.334711 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.334713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.334754 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:31:57.334815 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.334816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.334825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.334883 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.335363 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.336104 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.338127 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.338443 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.338501 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.339518 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.339520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.339520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.339526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.341292 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.341659 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.341660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.341660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.341706 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.341877 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.341877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.341878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.341988 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.342025 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.342178 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.342263 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:31:57.342865 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.334535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.0.1:56378)
0924 19:31:57.334695 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.334706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.334765 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.334917 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.334918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.335233 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.335290 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.335292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.337238 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.337238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.338043 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.338045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.338517 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.338518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.338582 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.339040 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.339044 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.339433 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.341369 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.341370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.341371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.341577 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.341765 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.341765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.341766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.341811 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.342059 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.342060 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.342092 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.342422 (+ 330us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.342754 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 19:31:57.350689 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.343850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.0.98.61:45165)
0924 19:31:57.343890 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.343892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.344035 (+ 143us) server_negotiation.cc:407] Connection header received
0924 19:31:57.344105 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.344106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.344116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.344166 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.344582 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.345130 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.346166 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.346358 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.346411 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.348449 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.348450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.348450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.348457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.349654 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.350000 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.350001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.350002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.350051 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.350213 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.350214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.350214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.350315 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.350367 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.350533 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.350600 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:31:57.350705 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.343809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.0.1:56390)
0924 19:31:57.343936 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.343977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.344043 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.344172 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.344175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.344459 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.344507 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.344509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.345210 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.345211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.346095 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.346098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.346420 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.346421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.346515 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.346908 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.346912 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.348354 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.349729 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.349736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.349738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.349936 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.350105 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.350105 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.350106 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.350164 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.350369 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.350376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.350429 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.350501 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.350616 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 19:31:57.351223 6483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45165: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6fb75f62318049e8b2d64c65559219d2"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45165
}
attrs {
promote: true
}
}
dest_uuid: "8b5cdc2713da469c9dcb55b7726d86ba"
cas_config_opid_index: -1
I20260924 19:31:57.351522 6483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } }
I20260924 19:31:57.352002 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6fb75f62318049e8b2d64c65559219d2 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } }
I20260924 19:31:57.352239 6990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:57.353116 6990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Refusing update from remote peer 8b5cdc2713da469c9dcb55b7726d86ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 19:31:57.353134 7044 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45165
I20260924 19:31:57.353209 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } }
I20260924 19:31:57.353273 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:57.353413 6948 master.cc:537] Master@127.0.98.61:45165 shutting down...
I20260924 19:31:57.353744 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:31:57.354449 6470 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 19:31:57.356132 6948 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:57.356238 6948 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:57.356288 6948 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: stopping tablet replica
W20260924 19:31:57.356999 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba -> Peer 6fb75f62318049e8b2d64c65559219d2 (127.0.98.61:45165): Couldn't send request to peer 6fb75f62318049e8b2d64c65559219d2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:31:57.369210 6948 master.cc:552] Master@127.0.98.61:45165 shutdown complete.
I20260924 19:31:57.369256 6948 master_runner.cc:305] Clearing existing system tablet
I20260924 19:31:57.369374 6948 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.369421 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.369966 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:31:57.370378 7049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.370522 6948 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.370595 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.370621 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
uuid: "6fb75f62318049e8b2d64c65559219d2"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1ddfdb7ed19562a285a8496e1722417"
server_key_iv: "30f47362536432e7930797d7600dc477"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.371383 6948 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:57.372964 6948 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:31:57.373035 6948 log.cc:1196] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 19:31:57.373238 6948 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Deleting consensus metadata
I20260924 19:31:57.373351 6948 master_runner.cc:315] Copying system tablet from 127.0.98.62:43727
I20260924 19:31:57.373974 6948 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43727
I20260924 19:31:57.379916 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.374327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.0.98.62:43727)
0924 19:31:57.374374 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.374376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.374412 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:31:57.374474 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.374475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.374483 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.374520 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.374891 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.375416 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.376461 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.376691 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.376748 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.377678 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.377679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.377679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.377686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.378954 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.379247 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.379248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.379249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.379301 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.379462 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.379463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.379463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.379565 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.379620 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.379753 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.379819 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:31:57.379927 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.374236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:40792)
0924 19:31:57.374344 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.374355 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.374406 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.374528 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.374528 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.374742 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.374814 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.374817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.375496 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.375497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.376389 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.376392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.376801 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.376802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.376859 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.377297 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.377301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.377590 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.379030 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.379031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.379031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.379186 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.379354 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.379355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.379355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.379413 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.379618 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.379619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.379664 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.379723 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.379819 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":59,"threads_started":1}
I20260924 19:31:57.380172 6504 tablet_copy_service.cc:140] P 8b5cdc2713da469c9dcb55b7726d86ba: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6fb75f62318049e8b2d64c65559219d2 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40792)
I20260924 19:31:57.380251 6504 tablet_copy_service.cc:161] P 8b5cdc2713da469c9dcb55b7726d86ba: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6fb75f62318049e8b2d64c65559219d2 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40792: session id = 6fb75f62318049e8b2d64c65559219d2-00000000000000000000000000000000
I20260924 19:31:57.380908 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:31:57.381516 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:31:57.382676 6948 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: tablet copy: Starting download of 0 data blocks...
I20260924 19:31:57.382823 6948 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: tablet copy: Starting download of 1 WAL segments...
I20260924 19:31:57.383628 6948 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:31:57.384487 6504 tablet_copy_service.cc:342] P 8b5cdc2713da469c9dcb55b7726d86ba: Request end of tablet copy session 6fb75f62318049e8b2d64c65559219d2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40792
I20260924 19:31:57.384558 6504 tablet_copy_service.cc:433] P 8b5cdc2713da469c9dcb55b7726d86ba: ending tablet copy session 6fb75f62318049e8b2d64c65559219d2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6fb75f62318049e8b2d64c65559219d2
I20260924 19:31:57.386658 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:57.387777 7057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.387840 7056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.388036 6948 server_base.cc:1034] running on GCE node
W20260924 19:31:57.388036 7059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.388195 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:57.388312 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:57.389425 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790278317389419 us; error 22 us; skew 500 ppm
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:57.390785 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 19:31:57.391043 6948 webserver.cc:460] Webserver started at http://127.0.98.61:33819/ using document root <none> and password file <none>
I20260924 19:31:57.391149 6948 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.391182 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.391814 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:31:57.392309 7066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.392436 6948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.392472 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
uuid: "6fb75f62318049e8b2d64c65559219d2"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1ddfdb7ed19562a285a8496e1722417"
server_key_iv: "30f47362536432e7930797d7600dc477"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.392524 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.411859 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:57.413460 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:57.416165 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45165
I20260924 19:31:57.416188 7118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45165 every 8 connection(s)
I20260924 19:31:57.416478 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:31:57.417400 7119 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:31:57.417910 7119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Bootstrap starting.
I20260924 19:31:57.420775 7119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:31:57.420998 7119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: Bootstrap complete.
I20260924 19:31:57.421177 7119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } }
I20260924 19:31:57.421275 7119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6fb75f62318049e8b2d64c65559219d2, State: Initialized, Role: LEARNER
I20260924 19:31:57.421321 7119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } }
I20260924 19:31:57.421509 7119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:57.421489 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } }
I20260924 19:31:57.421561 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:31:57.422976 7133 catalog_manager.cc:1259] Loaded cluster ID: af5bc9eb2aad45d0a00044e1d10d36dc
I20260924 19:31:57.423013 7133 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: loading cluster ID for follower catalog manager: success
I20260924 19:31:57.423980 7133 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: acquiring CA information for follower catalog manager: success
I20260924 19:31:57.424254 7133 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:31:57.432016 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.425432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.0.98.61:45165)
0924 19:31:57.425518 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.425520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.425547 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:31:57.425565 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.425565 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.425573 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.425616 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.426254 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.426843 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.427896 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.428088 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.428137 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.429626 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.429627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.429627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.429633 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.430938 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.431336 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.431337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.431337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.431389 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.431572 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.431572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.431573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.431661 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.431725 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.431852 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.431932 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":41,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 19:31:57.456460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39855
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43727,127.0.98.61:45165,127.0.98.60:39855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:31:57.536602 7137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:57.536839 7137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:57.536859 7137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:57.537972 7137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:31:57.538003 7137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:57.538017 7137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:57.538028 7137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:31:57.538038 7137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:31:57.539480 7137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43727,127.0.98.61:45165,127.0.98.60:39855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:57.539666 7137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:57.539903 7137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:57.544603 7143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.544697 7137 server_base.cc:1034] running on GCE node
W20260924 19:31:57.544652 7142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.544770 7145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.544986 7137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:57.545271 7137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:57.546445 7137 hybrid_clock.cc:648] HybridClock initialized: now 1790278317546429 us; error 29 us; skew 500 ppm
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:57.549052 7137 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 19:31:57.549459 7137 webserver.cc:460] Webserver started at http://127.0.98.60:45603/ using document root <none> and password file <none>
I20260924 19:31:57.549669 7137 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.549716 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.549834 7137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:31:57.550851 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data/instance:
uuid: "251bd89115e241ba9883c87e0db592fa"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1696865f53c936743a65ccbe7dcec3c"
server_key_iv: "cb5c475d15a8f5c5eef62ee42b5512d9"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.551167 7137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal/instance:
uuid: "251bd89115e241ba9883c87e0db592fa"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1696865f53c936743a65ccbe7dcec3c"
server_key_iv: "cb5c475d15a8f5c5eef62ee42b5512d9"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.552362 7137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:31:57.553053 7152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.553194 7137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.553272 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
uuid: "251bd89115e241ba9883c87e0db592fa"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1696865f53c936743a65ccbe7dcec3c"
server_key_iv: "cb5c475d15a8f5c5eef62ee42b5512d9"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.553334 7137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.563973 7137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:57.566066 7137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:57.566238 7137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:57.570037 7137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39855
I20260924 19:31:57.570077 7204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39855 every 8 connection(s)
I20260924 19:31:57.570406 7137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:31:57.570932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 7137
I20260924 19:31:57.571422 7205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:57.573465 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:31:57.581027 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.574246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.98.62:43727)
0924 19:31:57.574320 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.574322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.574332 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:31:57.574389 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.574391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.574400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.574447 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.574929 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.575437 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.576301 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.576519 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.576567 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.578726 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.578727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.578727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.578734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.579948 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.580274 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.580275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.580275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.580330 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.580486 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.580486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.580487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.580591 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.580678 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.580853 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.580936 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:31:57.581131 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.571704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.98.60:39855)
0924 19:31:57.571874 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.571882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.571909 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:31:57.572494 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.572502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.572556 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.572728 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.573408 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.574542 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.575359 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.575807 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.575879 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.578245 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.578252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.578252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.578258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.579753 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.580059 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.580062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.580062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.580136 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.580310 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.580311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.580312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.580535 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.580686 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.580777 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.580942 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260924 19:31:57.581143 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.574086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:40806)
0924 19:31:57.574226 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.574263 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.574329 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.574469 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.574472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.574769 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.574847 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.574849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.575523 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.575524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.576226 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.576228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.576597 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.576598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.576691 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.577080 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.577090 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.578641 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.580025 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.580031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.580033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.580207 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.580383 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.580384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.580384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.580435 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.580660 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.580666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.580745 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.580815 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.581052 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":122,"thread_start_us":70,"threads_started":1}
I20260924 19:31:57.581835 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 19:31:57.587841 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.582122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.0.98.61:45165)
0924 19:31:57.582159 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.582161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.582168 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:31:57.582191 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.582192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.582204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.582245 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.582438 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.582958 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.583689 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.583886 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.583945 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.585733 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.585734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.585735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.585741 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.586940 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.587241 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.587242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.587242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.587285 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.587478 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.587478 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.587478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.587571 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.587613 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.587717 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.587778 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:31:57.587846 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.582061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.0.1:56420)
0924 19:31:57.582099 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.582108 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.582139 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.582245 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.582246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.582349 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.582390 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.582391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.583047 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.583048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.583604 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.583606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.583970 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.583971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.584048 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.584421 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.584425 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.585650 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.587026 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.587026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.587027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.587180 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.587355 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.587356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.587356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.587416 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.587614 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.587614 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.587648 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.587699 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.587772 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:31:57.588290 7205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:31:57.594111 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.588516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.0.1:58248)
0924 19:31:57.588558 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.588568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.588615 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.588824 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.588824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.588941 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.588967 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.588969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.589582 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.589583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.590046 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.590048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.590383 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.590385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.590429 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.590773 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.590777 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.591877 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.593248 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.593249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.593249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.593427 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.593602 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.593603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.593603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.593650 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.593905 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.593905 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.593936 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.593980 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.594050 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 19:31:57.594178 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.588541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.0.98.60:39855)
0924 19:31:57.588567 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.588570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.588619 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:31:57.588760 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.588760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.588769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.588824 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.589025 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.589515 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.590111 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.590328 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.590378 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.591963 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.591964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.591964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.591970 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.593179 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.593489 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.593490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.593490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.593547 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.593733 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.593734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.593735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.593845 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.593910 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.594009 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.594097 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":14}
I20260924 19:31:57.595553 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Bootstrap starting.
I20260924 19:31:57.596331 7205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:57.596589 7205 log.cc:824] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:57.597296 7205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: No bootstrap required, opened a new log
I20260924 19:31:57.598695 7205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } }
I20260924 19:31:57.598834 7205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:57.598869 7205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 251bd89115e241ba9883c87e0db592fa, State: Initialized, Role: FOLLOWER
I20260924 19:31:57.598984 7205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } }
I20260924 19:31:57.599285 7205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:57.599249 7210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } } }
I20260924 19:31:57.599341 7210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:31:57.600921 7221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:31:57.600979 7221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:31:57.603343 7137 master_runner.cc:428] Detected that this master 251bd89115e241ba9883c87e0db592fa is joining an existing cluster
I20260924 19:31:57.603384 7137 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39855
I20260924 19:31:57.611341 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.604266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:40816)
0924 19:31:57.604427 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.604436 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.604977 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.605124 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.605124 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.605325 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.605408 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.605410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.606333 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.606334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.607106 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.607107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.607620 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.607622 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.607693 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.608254 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.608259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.608554 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.610330 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.610330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.610331 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.610543 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.610732 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.610733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.610733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.610787 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.611041 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.611041 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.611078 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.611133 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.611240 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260924 19:31:57.611613 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.604319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.0.98.62:43727)
0924 19:31:57.604420 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.604422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.604957 (+ 535us) server_negotiation.cc:407] Connection header received
0924 19:31:57.605039 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.605040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.605053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.605117 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.605491 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.606253 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.607193 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.607470 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.607511 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.608662 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.608664 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.608664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.608672 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.610250 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.610615 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.610616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.610616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.610674 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.610847 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.610848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.610849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.610969 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.611020 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.611452 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.611526 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278316, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:31:57.632406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 6546
I20260924 19:31:57.638039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38451
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43727
--webserver_port=43673
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165,127.0.98.60:39855 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:31:57.638190 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.604672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.0.1:58256)
0924 19:31:57.608714 (+ 4042us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.608732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.608796 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.608933 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.608934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.609096 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.609160 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.609162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.609984 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.609985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.610799 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.610800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.611205 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.611205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.611271 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.611712 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.611716 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.612054 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.637169 (+ 25115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.637169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.637170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.637398 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.637571 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.637572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.637572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.637617 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.637879 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.637879 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.637924 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.637981 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.638079 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4019,"thread_start_us":57,"threads_started":1}
I20260924 19:31:57.638468 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.604589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.98.60:39855)
0924 19:31:57.604621 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.604623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.608773 (+ 4150us) server_negotiation.cc:407] Connection header received
0924 19:31:57.608850 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.608851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.608858 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.608884 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.609231 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.609903 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.610867 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.611115 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.611148 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.612114 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.612115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.612115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.612121 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.637073 (+ 24952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.637474 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.637475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.637475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.637519 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.637669 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.637669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.637669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.637792 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.637859 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.638303 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.638375 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:31:57.640962 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.604311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.0.98.61:45165)
0924 19:31:57.604365 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.604367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.604953 (+ 586us) server_negotiation.cc:407] Connection header received
0924 19:31:57.605169 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.605170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.605180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.605230 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.605529 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.606020 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.607066 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.607354 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.607391 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.637857 (+ 30466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.637859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.637859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.637867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.639777 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.640178 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.640179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.640180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.640233 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.640428 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.640429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.640429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.640552 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.640590 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.640805 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.640880 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:31:57.641269 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.604373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.0.1:56436)
0924 19:31:57.604532 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.604545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.604978 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.605246 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.605247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.605402 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.605443 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.605445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.606092 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.606092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.606976 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.606978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.607969 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.607970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.608036 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.637333 (+ 29297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.637339 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.637752 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.639857 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.639857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.639858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.640095 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.640299 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.640300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.640300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.640361 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.640654 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.640655 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.640697 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.641069 (+ 372us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.641171 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260924 19:31:57.641950 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.614565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.0.1:58270)
0924 19:31:57.614609 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.614619 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.614653 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.614895 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.614896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.615054 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.615096 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.615098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.615703 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.615704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.616507 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.616509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.616970 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.616971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.617039 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.617403 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.617408 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.618628 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.640482 (+ 21854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.640483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.640484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.640734 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.640945 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.640945 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.640946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.641003 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.641673 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.641674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.641721 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.641766 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.641884 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:31:57.642289 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.614635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.0.98.60:39855)
0924 19:31:57.614775 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.614776 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.614782 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:31:57.614806 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.614807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.614815 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.614857 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.615165 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.615630 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.616606 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.616897 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.616957 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.618711 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.618712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.618712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.618718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.640387 (+ 21669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.640806 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.640807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.640808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.640873 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.641268 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.641269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.641269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.641598 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.642112 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.642149 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.642224 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260924 19:31:57.642608 6483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39855: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "251bd89115e241ba9883c87e0db592fa"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39855
}
attrs {
promote: true
}
}
dest_uuid: "8b5cdc2713da469c9dcb55b7726d86ba"
cas_config_opid_index: 5
I20260924 19:31:57.642863 6483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } }
I20260924 19:31:57.643181 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 251bd89115e241ba9883c87e0db592fa (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.643931 7045 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39855
I20260924 19:31:57.644006 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.644075 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:57.644361 7137 master.cc:537] Master@127.0.98.60:39855 shutting down...
W20260924 19:31:57.645126 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba -> Peer 251bd89115e241ba9883c87e0db592fa (127.0.98.60:39855): Couldn't send request to peer 251bd89115e241ba9883c87e0db592fa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:31:57.647812 7137 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:57.647907 7137 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:57.647946 7137 tablet_replica.cc:330] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: stopping tablet replica
I20260924 19:31:57.648612 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.644318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.0.1:56450)
0924 19:31:57.645283 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.645294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.645340 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.645522 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.645523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.645524 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:57.645568 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.645570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.646603 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.646604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.647460 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.647463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.648358 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.648359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.648425 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.648457 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.648506 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.648547 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948}
I20260924 19:31:57.648993 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.644367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.0.98.61:45165)
0924 19:31:57.644401 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.644402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.645348 (+ 946us) server_negotiation.cc:407] Connection header received
0924 19:31:57.645389 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.645390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.645413 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.645464 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:57.645894 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.646515 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.647542 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.648087 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.648151 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.648229 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.648896 (+ 667us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.648934 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:31:57.649183 7093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 LEARNER]: Refusing update from remote peer 8b5cdc2713da469c9dcb55b7726d86ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 19:31:57.649397 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 19:31:57.649966 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 251bd89115e241ba9883c87e0db592fa (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.650833 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.650906 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:31:57.651259 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b5cdc2713da469c9dcb55b7726d86ba. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: true } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.651338 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:31:57.651463 7043 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: attempting to promote NON_VOTER 6fb75f62318049e8b2d64c65559219d2 to VOTER
I20260924 19:31:57.651630 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } }
I20260924 19:31:57.652078 7093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 LEARNER]: Refusing update from remote peer 8b5cdc2713da469c9dcb55b7726d86ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 19:31:57.652118 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.651942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.0.1:58276)
0924 19:31:57.652060 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.652096 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260924 19:31:57.652217 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 19:31:57.652354 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.652294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.0.1:58290)
0924 19:31:57.652312 (+ 18us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.652334 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9,"threads_started":1}
W20260924 19:31:57.652423 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba -> Peer 251bd89115e241ba9883c87e0db592fa (127.0.98.60:39855): Couldn't send request to peer 251bd89115e241ba9883c87e0db592fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:31:57.652877 7045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6fb75f62318049e8b2d64c65559219d2 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.653023 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6fb75f62318049e8b2d64c65559219d2 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.653687 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.653753 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:57.653863 7045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.653920 7045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:57.653932 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.653992 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:31:57.661834 7137 master.cc:552] Master@127.0.98.60:39855 shutdown complete.
I20260924 19:31:57.661876 7137 master_runner.cc:305] Clearing existing system tablet
I20260924 19:31:57.661993 7137 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.662048 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.662636 7137 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.663050 7237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.663138 7137 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.663180 7137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.663203 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
uuid: "251bd89115e241ba9883c87e0db592fa"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1696865f53c936743a65ccbe7dcec3c"
server_key_iv: "cb5c475d15a8f5c5eef62ee42b5512d9"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.663791 7137 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:57.665048 7137 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:31:57.665107 7137 log.cc:1196] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 19:31:57.665267 7137 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Deleting consensus metadata
I20260924 19:31:57.665380 7137 master_runner.cc:315] Copying system tablet from 127.0.98.62:43727
I20260924 19:31:57.665925 7137 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43727
I20260924 19:31:57.671586 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.666310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.0.98.62:43727)
0924 19:31:57.666351 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.666352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.666362 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:31:57.666416 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.666417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.666426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.666470 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.666749 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.667236 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.667913 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.668112 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.668163 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.669372 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.669373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.669374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.669381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.670613 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.670918 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.670919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.670920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.670972 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.671131 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.671132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.671132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.671242 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.671297 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.671436 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.671498 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:31:57.671591 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.666189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:40824)
0924 19:31:57.666293 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.666303 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.666358 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.666472 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.666473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.666640 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.666677 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.666679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.667307 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.667308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.667835 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.667837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.668172 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.668173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.668235 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.669005 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.669010 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.669283 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.670689 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.670690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.670691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.670855 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.671025 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.671025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.671026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.671079 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.671299 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.671299 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.671343 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.671402 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.671500 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":67,"threads_started":1}
I20260924 19:31:57.671975 6504 tablet_copy_service.cc:140] P 8b5cdc2713da469c9dcb55b7726d86ba: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 251bd89115e241ba9883c87e0db592fa ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40824)
I20260924 19:31:57.672029 6504 tablet_copy_service.cc:161] P 8b5cdc2713da469c9dcb55b7726d86ba: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 251bd89115e241ba9883c87e0db592fa at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40824: session id = 251bd89115e241ba9883c87e0db592fa-00000000000000000000000000000000
I20260924 19:31:57.672479 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:31:57.672923 7137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:31:57.673852 7137 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: tablet copy: Starting download of 0 data blocks...
I20260924 19:31:57.673975 7137 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: tablet copy: Starting download of 1 WAL segments...
I20260924 19:31:57.674567 7137 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:31:57.675247 6504 tablet_copy_service.cc:342] P 8b5cdc2713da469c9dcb55b7726d86ba: Request end of tablet copy session 251bd89115e241ba9883c87e0db592fa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40824
I20260924 19:31:57.675295 6504 tablet_copy_service.cc:433] P 8b5cdc2713da469c9dcb55b7726d86ba: ending tablet copy session 251bd89115e241ba9883c87e0db592fa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 251bd89115e241ba9883c87e0db592fa
I20260924 19:31:57.679176 7137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:57.680253 7244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.680362 7247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.680269 7245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.680804 7137 server_base.cc:1034] running on GCE node
I20260924 19:31:57.680898 7137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:57.681016 7137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:57.682145 7137 hybrid_clock.cc:648] HybridClock initialized: now 1790278317682137 us; error 26 us; skew 500 ppm
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:57.683476 7137 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 19:31:57.683714 7137 webserver.cc:460] Webserver started at http://127.0.98.60:45655/ using document root <none> and password file <none>
I20260924 19:31:57.683822 7137 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.683871 7137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.684473 7137 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.684966 7254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.685060 7137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.685098 7137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
uuid: "251bd89115e241ba9883c87e0db592fa"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "c1696865f53c936743a65ccbe7dcec3c"
server_key_iv: "cb5c475d15a8f5c5eef62ee42b5512d9"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.685142 7137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.691809 7137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:57.693418 7137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:57.696081 7137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39855
I20260924 19:31:57.696424 7137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:31:57.696822 7306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39855 every 8 connection(s)
I20260924 19:31:57.697381 7307 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:31:57.697777 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Bootstrap starting.
I20260924 19:31:57.699895 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:31:57.700091 7307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Bootstrap complete.
I20260924 19:31:57.700273 7307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } }
I20260924 19:31:57.700351 7307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 251bd89115e241ba9883c87e0db592fa, State: Initialized, Role: LEARNER
I20260924 19:31:57.700387 7307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } }
I20260924 19:31:57.700570 7307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:31:57.700544 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:57.700807 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: This master's current role is: LEARNER
I20260924 19:31:57.701862 7321 catalog_manager.cc:1259] Loaded cluster ID: af5bc9eb2aad45d0a00044e1d10d36dc
I20260924 19:31:57.701960 7321 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: loading cluster ID for follower catalog manager: success
I20260924 19:31:57.703125 7321 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: acquiring CA information for follower catalog manager: success
I20260924 19:31:57.703385 7321 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:31:57.746855 7231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:57.747093 7231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:57.747121 7231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:57.748396 7231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:57.748448 7231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:57.748478 7231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:31:57.750511 7231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43673
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165,127.0.98.60:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:57.750773 7231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:57.751034 7231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:57.756009 7326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.756044 7327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.756191 7231 server_base.cc:1034] running on GCE node
W20260924 19:31:57.756248 7329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.756574 7231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:57.756896 7231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:57.758095 7231 hybrid_clock.cc:648] HybridClock initialized: now 1790278317758075 us; error 28 us; skew 500 ppm
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:57.761014 7231 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 19:31:57.761538 7231 webserver.cc:460] Webserver started at http://127.0.98.1:43673/ using document root <none> and password file <none>
I20260924 19:31:57.761796 7231 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.761863 7231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.763214 7231 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 19:31:57.763892 7336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.764075 7231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:31:57.764158 7231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "30a1e8e885b14651aac5c8f8d58ac786"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "7f69674d2d4da3561cbe40d5cc717c1c"
server_key_iv: "e42393148b9dfe6b044c4dc90b35c08a"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.764504 7231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.774243 7231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:57.776393 7231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:57.776525 7231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:57.776793 7231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:57.777083 7231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:57.777117 7231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.777143 7231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:57.777187 7231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.783084 7231 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38451
I20260924 19:31:57.783113 7449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38451 every 8 connection(s)
I20260924 19:31:57.783481 7231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:31:57.784178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 7231
I20260924 19:31:57.784325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 6680
I20260924 19:31:57.789618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39351
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43727
--webserver_port=44753
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165,127.0.98.60:39855 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:31:57.796125 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.784452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.98.1:42141)
0924 19:31:57.787199 (+ 2747us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.787206 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.787268 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.787421 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.787421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.787613 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.788182 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.788184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.789090 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.789091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.790058 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.790060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.790514 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.790648 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.791185 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.791194 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.792903 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.794559 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.794565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.794567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.794843 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.795057 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.795059 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.795059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.795185 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.795461 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.795468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.795621 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.795744 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.795902 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2732,"mutex_wait_us":484,"thread_start_us":53,"threads_started":1}
I20260924 19:31:57.796419 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.786769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42141 (local address 127.0.98.61:45165)
0924 19:31:57.786819 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.786822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.787242 (+ 420us) server_negotiation.cc:407] Connection header received
0924 19:31:57.787328 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.787329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.787339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.787367 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.788267 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.789001 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790145 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.790401 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790437 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.792981 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.792982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.792983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.792990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.794485 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.794929 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.794931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.794932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.794989 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.795243 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.795244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.795245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.795377 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.795434 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.795784 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.795862 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:31:57.797170 7452 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45165
I20260924 19:31:57.797266 7452 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:57.797449 7452 heartbeater.cc:511] Master 127.0.98.61:45165 requested a full tablet report, sending...
I20260924 19:31:57.797935 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.784162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45073 (local address 127.0.98.62:43727)
0924 19:31:57.784209 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.784211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.787134 (+ 2923us) server_negotiation.cc:407] Connection header received
0924 19:31:57.787168 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.787169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.787181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.787235 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.787957 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.788744 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790150 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.790424 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790460 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.795163 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.795165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.795165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.795174 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.796698 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.797030 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.797032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.797032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.797086 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.797364 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.797365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.797365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.797493 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.797532 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.797781 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.797853 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:31:57.797941 7083 ts_manager.cc:194] Registered new tserver with Master: 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451)
I20260924 19:31:57.798446 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.784192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:45073)
0924 19:31:57.784357 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.787069 (+ 2712us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.787125 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.787290 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.787291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.787575 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.787851 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.787856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.788858 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.788859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.790059 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.790062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.790527 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.790648 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.791229 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.791234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.794342 (+ 3108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.796759 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.796760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.796761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.796949 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.797156 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.797156 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.797157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.797208 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.797620 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.797620 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.797659 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.798240 (+ 581us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.798335 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":8,"thread_start_us":84,"threads_started":1}
I20260924 19:31:57.798724 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.784196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.98.1:41773)
0924 19:31:57.785785 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.785826 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.786424 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.786805 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.786810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.787605 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.787851 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.787856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.789052 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.789053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.790285 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.790286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.790729 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790729 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.790796 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.791495 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.791500 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.794612 (+ 3112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.797603 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.797603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.797604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.797797 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.797987 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.797988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.797988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.798042 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.798302 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.798303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.798350 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.798406 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.798495 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1565,"mutex_wait_us":6,"thread_start_us":82,"threads_started":1}
I20260924 19:31:57.798797 7451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43727
I20260924 19:31:57.798839 7451 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:57.798909 7451 heartbeater.cc:511] Master 127.0.98.62:43727 requested a full tablet report, sending...
I20260924 19:31:57.799026 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.784908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41773 (local address 127.0.98.60:39855)
0924 19:31:57.785059 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.785061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.785998 (+ 937us) server_negotiation.cc:407] Connection header received
0924 19:31:57.786522 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.786523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.786534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.786711 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.787951 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.788979 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790355 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.790628 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.790680 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.794689 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.794690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.794690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.794697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.797531 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.797864 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.797865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.797866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.797927 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.798102 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.798103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.798103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.798228 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.798280 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.798877 (+ 597us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.798943 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":58,"threads_started":1}
I20260924 19:31:57.799176 6483 ts_manager.cc:194] Re-registered known tserver with Master: 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451)
I20260924 19:31:57.799288 7450 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39855
I20260924 19:31:57.799335 7450 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:57.799392 7450 heartbeater.cc:511] Master 127.0.98.60:39855 requested a full tablet report, sending...
I20260924 19:31:57.799641 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45073
I20260924 19:31:57.799681 7271 ts_manager.cc:194] Registered new tserver with Master: 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451)
W20260924 19:31:57.871340 7458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:57.871551 7458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:57.871580 7458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:57.872834 7458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:57.872895 7458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:57.872931 7458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:31:57.874868 7458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44753
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165,127.0.98.60:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:57.875128 7458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:57.875351 7458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:57.880244 7466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.880379 7458 server_base.cc:1034] running on GCE node
W20260924 19:31:57.880311 7463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:57.880343 7464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:57.880703 7458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:57.880918 7458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:57.882055 7458 hybrid_clock.cc:648] HybridClock initialized: now 1790278317882042 us; error 30 us; skew 500 ppm
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:57.884456 7458 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 19:31:57.884856 7458 webserver.cc:460] Webserver started at http://127.0.98.2:44753/ using document root <none> and password file <none>
I20260924 19:31:57.885056 7458 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:57.885143 7458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:57.886415 7458 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:57.886916 7473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.887140 7458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:31:57.887223 7458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "a77223cab72e427ba7553a4480fe9d93"
format_stamp: "Formatted at 2026-09-24 19:31:56 on dist-test-slave-m8jb"
server_key: "f7f75c39e802ac70548b7a1af15aec4b"
server_key_iv: "4b35cf5cd93bed3fd3ded23ee38d2615"
server_key_version: "encryptionkey@0"
I20260924 19:31:57.887545 7458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:57.899369 7458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:57.901094 7458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:57.901207 7458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:57.901407 7458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:57.901685 7458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:57.901722 7458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.901741 7458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:57.901757 7458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:57.907086 7458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39351
I20260924 19:31:57.907151 7586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39351 every 8 connection(s)
I20260924 19:31:57.907433 7458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:31:57.914316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 7458
I20260924 19:31:57.914477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 6814
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 19:31:57 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278317, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 19:31:57.918895 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.908427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41401 (local address 127.0.98.61:45165)
0924 19:31:57.908488 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.908490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.908499 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:57.909478 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.909479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.909491 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.909537 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.910340 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.911106 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.912894 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.913103 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.913160 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.915547 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.915549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.915549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.915556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.917400 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.917844 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.917845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.917846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.917914 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.918183 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.918184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.918184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.918312 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.918351 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.918714 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.918791 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:31:57.918970 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.908128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34843 (local address 127.0.98.60:39855)
0924 19:31:57.908173 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.908175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.908381 (+ 206us) server_negotiation.cc:407] Connection header received
0924 19:31:57.909708 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.909710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.909772 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.909826 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.910366 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.910857 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.912194 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.912513 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.912582 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.916239 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.916240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.916241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.916247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.917824 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.918227 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.918228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.918229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.918291 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.918477 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.918478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.918478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.918592 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.918628 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.918800 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.918883 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:31:57.919073 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.908067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51491 (local address 127.0.98.62:43727)
0924 19:31:57.908118 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:57.908121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:57.908381 (+ 260us) server_negotiation.cc:407] Connection header received
0924 19:31:57.909595 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:57.909596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:57.909607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:57.909657 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:57.910375 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.911106 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.912194 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.912513 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.912582 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.915975 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:57.915976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:57.915977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:57.915984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:57.917960 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.918310 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.918311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.918311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.918363 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:57.918569 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:57.918570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:57.918570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:57.918690 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:57.918728 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:57.918923 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 19:31:57.918993 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:31:57.919420 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.908077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.98.2:34843)
0924 19:31:57.908240 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.908301 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.909379 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.909859 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.909860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.910080 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.910255 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.910261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.910950 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.910951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.912104 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.912106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.912594 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.912595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.912759 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.913371 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.913375 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.916157 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.917909 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.917910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.917910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.918153 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.918353 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.918353 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.918354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.918406 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.918666 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.918666 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.918706 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.919039 (+ 333us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.919240 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":104,"thread_start_us":87,"threads_started":1}
I20260924 19:31:57.919420 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.908334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.98.2:41401)
0924 19:31:57.908433 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.908441 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.909379 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.909631 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.909636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.910048 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.910254 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.910259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.912196 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.912196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.912778 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.912780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.913167 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.913168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.913232 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.913490 (+ 258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.913494 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.915462 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.917484 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.917491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.917493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.917760 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.917984 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.917984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.917985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.918109 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.918393 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.918400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.918571 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.918988 (+ 417us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.919166 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260924 19:31:57.920058 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:57.908077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:51491)
0924 19:31:57.908304 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:57.908312 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:57.909368 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:57.909690 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:57.909691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:57.910064 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:57.910255 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.910261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.911185 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.911186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.912104 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:57.912106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:57.912687 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:57.912688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:57.912760 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:57.913322 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:57.913332 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:57.915901 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:57.918032 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.918032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.918033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.918238 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.918452 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:57.918453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:57.918453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:57.918505 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:57.918768 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:57.918769 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:57.918819 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 19:31:57.919888 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0924 19:31:57.919985 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":28,"thread_start_us":86,"threads_started":1}
I20260924 19:31:57.920463 7588 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43727
I20260924 19:31:57.920559 7588 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:57.920769 7588 heartbeater.cc:511] Master 127.0.98.62:43727 requested a full tablet report, sending...
I20260924 19:31:57.920985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41535
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43727
--webserver_port=36059
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165,127.0.98.60:39855 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:31:57.921129 7589 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45165
I20260924 19:31:57.921168 7589 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:57.921154 6483 ts_manager.cc:194] Re-registered known tserver with Master: a77223cab72e427ba7553a4480fe9d93 (127.0.98.2:39351)
I20260924 19:31:57.921221 7589 heartbeater.cc:511] Master 127.0.98.61:45165 requested a full tablet report, sending...
I20260924 19:31:57.921504 7083 ts_manager.cc:194] Registered new tserver with Master: a77223cab72e427ba7553a4480fe9d93 (127.0.98.2:39351)
I20260924 19:31:57.921646 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51491
I20260924 19:31:57.922161 7587 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39855
I20260924 19:31:57.922217 7587 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:57.922590 7587 heartbeater.cc:511] Master 127.0.98.60:39855 requested a full tablet report, sending...
I20260924 19:31:57.922873 7271 ts_manager.cc:194] Registered new tserver with Master: a77223cab72e427ba7553a4480fe9d93 (127.0.98.2:39351)
W20260924 19:31:58.001631 7594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:31:58.001848 7594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:31:58.001873 7594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:31:58.003072 7594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:31:58.003113 7594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:31:58.003137 7594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:31:58.004587 7594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36059
--tserver_master_addrs=127.0.98.62:43727,127.0.98.61:45165,127.0.98.60:39855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:31:58.004833 7594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:31:58.005055 7594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:31:58.009773 7599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:58.009783 7600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:31:58.009810 7602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:31:58.010083 7594 server_base.cc:1034] running on GCE node
I20260924 19:31:58.010273 7594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:31:58.010516 7594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:31:58.011667 7594 hybrid_clock.cc:648] HybridClock initialized: now 1790278318011651 us; error 27 us; skew 500 ppm
Sep 24 19:31:58 dist-test-slave-m8jb krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278318, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:31:58.014155 7594 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 19:31:58.014602 7594 webserver.cc:460] Webserver started at http://127.0.98.3:36059/ using document root <none> and password file <none>
I20260924 19:31:58.014824 7594 fs_manager.cc:359] Metadata directory not provided
I20260924 19:31:58.014879 7594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:31:58.016216 7594 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:31:58.016753 7609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:58.016897 7594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:58.016978 7594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "1d22290598ca4676a43fd50dc9cfc77e"
format_stamp: "Formatted at 2026-09-24 19:31:57 on dist-test-slave-m8jb"
server_key: "670df4ccfed6bd2807c4325c1d93136a"
server_key_iv: "fe0dc3613460a14def9f342789d90092"
server_key_version: "encryptionkey@0"
I20260924 19:31:58.017304 7594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:31:58.031939 7594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:31:58.034006 7594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:31:58.034113 7594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:31:58.034334 7594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:31:58.034615 7594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:31:58.034646 7594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:58.034667 7594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:31:58.034689 7594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:31:58.039819 7594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41535
I20260924 19:31:58.039868 7722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41535 every 8 connection(s)
I20260924 19:31:58.040172 7594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:31:58.045569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 7594
Sep 24 19:31:58 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278318, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 19:31:58 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278318, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 19:31:58 dist-test-slave-m8jb krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278318, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:31:58.050807 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.041253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36127 (local address 127.0.98.60:39855)
0924 19:31:58.041294 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.041296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.041376 (+ 80us) server_negotiation.cc:407] Connection header received
0924 19:31:58.041678 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.041679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.041690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.041744 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.042644 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.043410 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.044482 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.044721 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.044781 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.047570 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.047572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.047572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.047579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.049351 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.049780 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.049781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.049782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.049837 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.050102 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.050103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.050103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.050223 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.050275 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.050636 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.050717 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:58.050930 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.040862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.98.3:43209)
0924 19:31:58.041016 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.041051 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.041592 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.041824 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.041829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.042341 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:58.042958 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.042960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.043564 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.043564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.044558 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.044561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.044960 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.044961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.045025 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.045490 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:58.045495 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:58.047763 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:58.049814 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:58.049814 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:58.049815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:58.049982 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:58.050155 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:58.050155 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:58.050156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:58.050203 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:58.050455 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:58.050456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:58.050489 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.050553 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.050717 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":37,"thread_start_us":87,"threads_started":1}
I20260924 19:31:58.051285 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.041133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.98.3:36127)
0924 19:31:58.041319 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.041329 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.041599 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.042092 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.042092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.042343 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:58.042548 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.042552 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.043511 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.043512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.044403 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.044405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.044794 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.044795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.044920 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.045359 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:58.045368 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:58.047493 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:58.049437 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:58.049442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:58.049444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:58.049705 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:58.049913 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:58.049913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:58.049914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:58.050041 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:58.050298 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:58.050304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:58.050456 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.050553 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.051198 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":567,"thread_start_us":57,"threads_started":1}
I20260924 19:31:58.051640 6524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.040899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42419 (local address 127.0.98.62:43727)
0924 19:31:58.040942 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.040944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.041300 (+ 356us) server_negotiation.cc:407] Connection header received
0924 19:31:58.041709 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.041710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.041722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.041752 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.042708 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.043442 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.044687 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.044953 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.045015 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.048492 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.048494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.048494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.048501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.050219 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.050561 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.050562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.050562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.050620 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.051097 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.051098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.051099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.051219 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.051261 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.051472 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.051551 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:31:58.051718 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.040845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43209 (local address 127.0.98.61:45165)
0924 19:31:58.040899 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.040901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.041133 (+ 232us) server_negotiation.cc:407] Connection header received
0924 19:31:58.041708 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.041710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.041722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.041752 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.043027 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.043501 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.044661 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.044900 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.044951 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.048058 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.048059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.048060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.048067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.049745 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.050049 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.050049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.050050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.050097 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.050260 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.050261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.050261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.050381 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.050433 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.051558 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.051630 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:58.051787 7723 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39855
I20260924 19:31:58.051893 7723 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:58.051964 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.040865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:42419)
0924 19:31:58.041233 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.041242 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.041594 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.041829 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.041830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.042341 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:31:58.042549 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.042559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.043517 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.043517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.044562 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.044563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.045065 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.045065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.045122 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.047079 (+ 1957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:31:58.047084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:31:58.048398 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:31:58.050287 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:58.050288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:58.050288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:58.050487 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:58.050688 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:31:58.050689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:31:58.050689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:31:58.051018 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:31:58.051305 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:31:58.051306 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:31:58.051362 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.051765 (+ 403us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.051858 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":141,"thread_start_us":83,"threads_started":1}
I20260924 19:31:58.052081 7723 heartbeater.cc:511] Master 127.0.98.60:39855 requested a full tablet report, sending...
I20260924 19:31:58.052270 7724 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43727
I20260924 19:31:58.052318 7724 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:58.052371 7725 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45165
I20260924 19:31:58.052387 7724 heartbeater.cc:511] Master 127.0.98.62:43727 requested a full tablet report, sending...
I20260924 19:31:58.052407 7725 heartbeater.cc:464] Registering TS with master...
I20260924 19:31:58.052467 7725 heartbeater.cc:511] Master 127.0.98.61:45165 requested a full tablet report, sending...
I20260924 19:31:58.052563 7271 ts_manager.cc:194] Registered new tserver with Master: 1d22290598ca4676a43fd50dc9cfc77e (127.0.98.3:41535)
I20260924 19:31:58.052768 7083 ts_manager.cc:194] Registered new tserver with Master: 1d22290598ca4676a43fd50dc9cfc77e (127.0.98.3:41535)
I20260924 19:31:58.052913 6483 ts_manager.cc:194] Re-registered known tserver with Master: 1d22290598ca4676a43fd50dc9cfc77e (127.0.98.3:41535)
I20260924 19:31:58.053367 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42419
I20260924 19:31:58.055949 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.047072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.0.98.62:43727)
0924 19:31:58.047217 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.047218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.047226 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:58.047245 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.047245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.047257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.047335 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.048379 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.049120 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.050157 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.050445 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.050741 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.052868 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.052869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.052870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.052876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.054420 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.055016 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.055017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.055017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.055075 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.055262 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.055263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.055263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.055379 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.055468 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.055770 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.055858 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260924 19:31:58.057415 6483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:31:58.059367 6483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:31:58.060673 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.046937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.0.98.60:39855)
0924 19:31:58.047033 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.047034 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.050887 (+ 3853us) server_negotiation.cc:407] Connection header received
0924 19:31:58.053658 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.053659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.053670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.053704 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.054923 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.055627 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.056569 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.056879 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.056936 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.057991 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.057992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.057992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.057999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.059292 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.059622 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.059623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.059623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.059674 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.059902 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.059903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.059903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.060050 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.060155 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.060487 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.060562 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260924 19:31:58.060904 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.046746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.0.98.61:45165)
0924 19:31:58.051786 (+ 5040us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.051788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.051795 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:31:58.051813 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.051814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.051823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.051884 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.052841 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.053331 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.054157 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.054346 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.054401 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.055544 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.055545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.055546 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.055552 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.057008 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.060018 (+ 3010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.060020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.060020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.060075 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.060261 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.060261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.060262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.060405 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.060469 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.060676 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.060741 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5026,"thread_start_us":66,"threads_started":1}
I20260924 19:31:58.066834 7042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.063127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.0.1:54814)
0924 19:31:58.063238 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.063247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.063279 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.063932 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.063933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.063934 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:58.063967 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.063968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.065245 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.065246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.065945 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.065946 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.066615 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.066616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.066673 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.066676 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.066729 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.066770 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 19:31:58.066857 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.063115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.0.98.3:41535)
0924 19:31:58.063264 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.063269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.063546 (+ 277us) server_negotiation.cc:407] Connection header received
0924 19:31:58.063636 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.063638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.063652 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.063718 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:58.063824 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.064459 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.065400 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.066382 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.066468 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.066542 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.066718 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.066762 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260924 19:31:58.066864 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.063367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.0.98.2:39351)
0924 19:31:58.063502 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.063506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.063515 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:58.063547 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.063549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.063563 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.064035 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:58.064092 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.065168 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.066021 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.066547 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.066608 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.066683 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.066744 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.066793 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260924 19:31:58.067070 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.063035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41535 (local address 127.0.0.1:58954)
0924 19:31:58.063487 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.063496 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.063572 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.063718 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.063719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.063720 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:58.063770 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.063772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.064530 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.064531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.065321 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.065323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.066573 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.066574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.066630 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.066635 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.066964 (+ 329us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.067003 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":326,"thread_start_us":65,"threads_started":1}
I20260924 19:31:58.067127 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.063216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.0.98.1:38451)
0924 19:31:58.063361 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.063364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.063372 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:58.063403 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.063405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.063417 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.063529 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:58.063661 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.064740 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.065916 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.066615 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.066872 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.066994 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.067018 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.067056 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260924 19:31:58.067524 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.063035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:58472)
0924 19:31:58.063077 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.063086 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.063126 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.063533 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.063534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.063536 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:58.063585 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.063587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.064826 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.064827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.065828 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.065831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.066700 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.066701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.066751 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.066754 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.066779 (+ 25us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.066818 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 19:31:58.067718 7521 tablet_service.cc:1467] Processing CreateTablet for tablet afa79a8c8b1f441db5b8231559a5cfc9 (DEFAULT_TABLE table=test-table [id=e996ed2aa3a74907b76f88eb8badb124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:58.068104 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa79a8c8b1f441db5b8231559a5cfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:58.068156 7384 tablet_service.cc:1467] Processing CreateTablet for tablet afa79a8c8b1f441db5b8231559a5cfc9 (DEFAULT_TABLE table=test-table [id=e996ed2aa3a74907b76f88eb8badb124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:58.068411 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa79a8c8b1f441db5b8231559a5cfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:58.069075 7657 tablet_service.cc:1467] Processing CreateTablet for tablet afa79a8c8b1f441db5b8231559a5cfc9 (DEFAULT_TABLE table=test-table [id=e996ed2aa3a74907b76f88eb8badb124]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:31:58.069396 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afa79a8c8b1f441db5b8231559a5cfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:31:58.071090 7748 tablet_bootstrap.cc:492] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: Bootstrap starting.
I20260924 19:31:58.071344 7749 tablet_bootstrap.cc:492] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: Bootstrap starting.
I20260924 19:31:58.072151 7749 tablet_bootstrap.cc:654] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:58.072160 7748 tablet_bootstrap.cc:654] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:58.072455 7748 log.cc:824] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:58.072494 7749 log.cc:824] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:58.072726 7750 tablet_bootstrap.cc:492] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: Bootstrap starting.
I20260924 19:31:58.073266 7748 tablet_bootstrap.cc:492] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: No bootstrap required, opened a new log
I20260924 19:31:58.073320 7750 tablet_bootstrap.cc:654] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: Neither blocks nor log segments found. Creating new log.
I20260924 19:31:58.073338 7748 ts_tablet_manager.cc:1397] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:31:58.073520 7749 tablet_bootstrap.cc:492] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: No bootstrap required, opened a new log
I20260924 19:31:58.073585 7749 ts_tablet_manager.cc:1397] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:31:58.073582 7750 log.cc:824] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:31:58.074743 7750 tablet_bootstrap.cc:492] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: No bootstrap required, opened a new log
I20260924 19:31:58.074800 7750 ts_tablet_manager.cc:1397] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:31:58.075793 7748 raft_consensus.cc:348] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.075793 7749 raft_consensus.cc:348] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.075953 7748 raft_consensus.cc:374] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:58.075953 7749 raft_consensus.cc:374] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:58.075987 7749 raft_consensus.cc:729] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77223cab72e427ba7553a4480fe9d93, State: Initialized, Role: FOLLOWER
I20260924 19:31:58.075987 7748 raft_consensus.cc:729] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a1e8e885b14651aac5c8f8d58ac786, State: Initialized, Role: FOLLOWER
I20260924 19:31:58.076100 7749 consensus_queue.cc:260] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.076141 7748 consensus_queue.cc:260] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.076340 7749 ts_tablet_manager.cc:1428] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:31:58.076340 7748 ts_tablet_manager.cc:1428] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:31:58.076613 7588 heartbeater.cc:503] Master 127.0.98.62:43727 was elected leader, sending a full tablet report...
I20260924 19:31:58.076620 7451 heartbeater.cc:503] Master 127.0.98.62:43727 was elected leader, sending a full tablet report...
I20260924 19:31:58.077841 7750 raft_consensus.cc:348] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.077991 7750 raft_consensus.cc:374] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:31:58.078007 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.074346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.0.98.60:39855)
0924 19:31:58.074383 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.074385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.074393 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:58.074454 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.074455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.074476 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.074525 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:58.074679 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.075402 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.077100 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.077619 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.077679 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.077749 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.077925 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.077961 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:31:58.078016 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.074299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.0.1:58298)
0924 19:31:58.074348 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.074358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.074395 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.074570 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.074571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.074572 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:58.074611 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.074613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.075537 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.075538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.077019 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.077023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.077779 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.077780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.077853 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.077857 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.077913 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.077952 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:31:58.078022 7750 raft_consensus.cc:729] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d22290598ca4676a43fd50dc9cfc77e, State: Initialized, Role: FOLLOWER
I20260924 19:31:58.078393 7276 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 19:31:58.079133 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b5cdc2713da469c9dcb55b7726d86ba. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:58.079216 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: This master's current role is: LEARNER
I20260924 19:31:58.079468 7750 consensus_queue.cc:260] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.079741 7724 heartbeater.cc:503] Master 127.0.98.62:43727 was elected leader, sending a full tablet report...
I20260924 19:31:58.080021 7750 ts_tablet_manager.cc:1428] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260924 19:31:58.105896 7759 raft_consensus.cc:482] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:31:58.106015 7759 raft_consensus.cc:504] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.106309 7759 leader_election.cc:288] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451), a77223cab72e427ba7553a4480fe9d93 (127.0.98.2:39351)
I20260924 19:31:58.110102 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.106646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40395 (local address 127.0.98.2:39351)
0924 19:31:58.106690 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.106692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.106699 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:31:58.106722 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.106723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.106734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.106780 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:58.106935 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.107643 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.109166 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.109741 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.109819 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.109829 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.109990 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.110029 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:31:58.110102 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.106534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35349 (local address 127.0.98.1:38451)
0924 19:31:58.106590 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.106591 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.106600 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:58.106656 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.106657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.106668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.106725 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:58.106903 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.107666 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.109166 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.109762 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.109819 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.109829 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.109983 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.110026 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:31:58.110116 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.106466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.98.3:35349)
0924 19:31:58.106523 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.106535 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.106587 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.106739 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.106740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.106743 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:58.106830 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.106832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.107722 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.107722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.109079 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.109081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.109822 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.109822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.109908 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.109913 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.109971 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.110023 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":126}
I20260924 19:31:58.110126 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.106564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.98.3:40395)
0924 19:31:58.106608 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:58.106617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:58.106653 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:58.106788 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:58.106788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:58.106790 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:58.106857 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.106858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.107715 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.107717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.109079 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:58.109081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.109821 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.109822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:58.109924 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.109927 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:58.109969 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:31:58.110024 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":32}
I20260924 19:31:58.110479 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa79a8c8b1f441db5b8231559a5cfc9" candidate_uuid: "1d22290598ca4676a43fd50dc9cfc77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77223cab72e427ba7553a4480fe9d93" is_pre_election: true
I20260924 19:31:58.110479 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa79a8c8b1f441db5b8231559a5cfc9" candidate_uuid: "1d22290598ca4676a43fd50dc9cfc77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a1e8e885b14651aac5c8f8d58ac786" is_pre_election: true
I20260924 19:31:58.110627 7404 raft_consensus.cc:2454] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d22290598ca4676a43fd50dc9cfc77e in term 0.
I20260924 19:31:58.110627 7541 raft_consensus.cc:2454] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d22290598ca4676a43fd50dc9cfc77e in term 0.
I20260924 19:31:58.110850 7610 leader_election.cc:302] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d22290598ca4676a43fd50dc9cfc77e, a77223cab72e427ba7553a4480fe9d93; no voters:
I20260924 19:31:58.110973 7759 raft_consensus.cc:2781] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:31:58.111042 7759 raft_consensus.cc:482] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:31:58.111064 7759 raft_consensus.cc:3037] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:58.111737 7759 raft_consensus.cc:504] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.111852 7759 leader_election.cc:288] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [CANDIDATE]: Term 1 election: Requested vote from peers 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451), a77223cab72e427ba7553a4480fe9d93 (127.0.98.2:39351)
I20260924 19:31:58.112035 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa79a8c8b1f441db5b8231559a5cfc9" candidate_uuid: "1d22290598ca4676a43fd50dc9cfc77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a1e8e885b14651aac5c8f8d58ac786"
I20260924 19:31:58.112102 7541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afa79a8c8b1f441db5b8231559a5cfc9" candidate_uuid: "1d22290598ca4676a43fd50dc9cfc77e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a77223cab72e427ba7553a4480fe9d93"
I20260924 19:31:58.112118 7404 raft_consensus.cc:3037] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:58.112185 7541 raft_consensus.cc:3037] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:31:58.113127 7541 raft_consensus.cc:2454] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d22290598ca4676a43fd50dc9cfc77e in term 1.
I20260924 19:31:58.113284 7610 leader_election.cc:302] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d22290598ca4676a43fd50dc9cfc77e, a77223cab72e427ba7553a4480fe9d93; no voters:
I20260924 19:31:58.113328 7404 raft_consensus.cc:2454] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d22290598ca4676a43fd50dc9cfc77e in term 1.
I20260924 19:31:58.113402 7759 raft_consensus.cc:2781] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:31:58.113565 7759 raft_consensus.cc:686] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 LEADER]: Becoming Leader. State: Replica: 1d22290598ca4676a43fd50dc9cfc77e, State: Running, Role: LEADER
I20260924 19:31:58.113655 7759 consensus_queue.cc:237] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } }
I20260924 19:31:58.114485 6483 catalog_manager.cc:5495] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d22290598ca4676a43fd50dc9cfc77e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1d22290598ca4676a43fd50dc9cfc77e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } health_report { overall_health: UNKNOWN } } }
I20260924 19:31:58.114979 7235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: true } } }
I20260924 19:31:58.115068 7235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:58.115723 7045 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: attempting to promote NON_VOTER 251bd89115e241ba9883c87e0db592fa to VOTER
I20260924 19:31:58.115832 7045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } }
I20260924 19:31:58.116300 7276 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 LEARNER]: Refusing update from remote peer 8b5cdc2713da469c9dcb55b7726d86ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 19:31:58.116338 7093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Refusing update from remote peer 8b5cdc2713da469c9dcb55b7726d86ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 19:31:58.116501 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 19:31:58.116595 7235 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 19:31:58.117147 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 251bd89115e241ba9883c87e0db592fa (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:31:58.117264 7093 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 251bd89115e241ba9883c87e0db592fa (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:31:58.117463 7276 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 251bd89115e241ba9883c87e0db592fa (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:31:58.117903 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:31:58.117969 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:58.118019 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:31:58.118057 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:58.118039 7235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:31:58.118094 7235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba [sys.catalog]: This master's current role is: LEADER
I20260924 19:31:58.118127 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:31:58.118196 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:31:58.118393 7310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:31:58.118479 7310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:31:58.123620 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.121115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.0.98.3:41535)
0924 19:31:58.121165 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.121167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.121176 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:31:58.121232 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.121233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.121248 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.121306 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:31:58.121419 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.121998 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.122914 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.123124 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.123176 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.123349 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:31:58.123439 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:31:58.123492 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.123552 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.123568 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:31:58.125478 7404 raft_consensus.cc:1261] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 1 FOLLOWER]: Refusing update from remote peer 1d22290598ca4676a43fd50dc9cfc77e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:58.125478 7541 raft_consensus.cc:1261] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 1 FOLLOWER]: Refusing update from remote peer 1d22290598ca4676a43fd50dc9cfc77e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:31:58.125814 7760 consensus_queue.cc:1035] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:58.125929 7759 consensus_queue.cc:1035] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:31:58.128348 7763 mvcc.cc:205] Tried to move back new op lower bound from 7332979991039545344 to 7332979990994223104. Current Snapshot: MvccSnapshot[applied={T|T < 7332979991039545344}]
I20260924 19:31:58.128758 7765 mvcc.cc:205] Tried to move back new op lower bound from 7332979991039545344 to 7332979990994223104. Current Snapshot: MvccSnapshot[applied={T|T < 7332979991039545344}]
I20260924 19:31:58.130601 6483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:58.130692 6483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:31:58.131969 6483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b5cdc2713da469c9dcb55b7726d86ba: Sending DeleteTablet for 3 replicas of tablet afa79a8c8b1f441db5b8231559a5cfc9
I20260924 19:31:58.132229 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet afa79a8c8b1f441db5b8231559a5cfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58954
I20260924 19:31:58.132270 7384 tablet_service.cc:1514] Processing DeleteTablet for tablet afa79a8c8b1f441db5b8231559a5cfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58472
I20260924 19:31:58.132462 7774 tablet_replica.cc:330] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: stopping tablet replica
I20260924 19:31:58.132558 7774 raft_consensus.cc:2229] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:58.132687 7774 raft_consensus.cc:2258] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:58.132843 7773 tablet_replica.cc:330] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: stopping tablet replica
I20260924 19:31:58.132941 7773 raft_consensus.cc:2229] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:31:58.133033 7773 raft_consensus.cc:2258] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:58.133141 7521 tablet_service.cc:1514] Processing DeleteTablet for tablet afa79a8c8b1f441db5b8231559a5cfc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:31:58 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54814
I20260924 19:31:58.133493 7774 ts_tablet_manager.cc:1905] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:58.133721 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 6452
I20260924 19:31:58.134382 7773 ts_tablet_manager.cc:1905] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:58.136185 7773 ts_tablet_manager.cc:1918] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:58.136310 7773 log.cc:1196] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/afa79a8c8b1f441db5b8231559a5cfc9
I20260924 19:31:58.136613 7773 ts_tablet_manager.cc:1939] T afa79a8c8b1f441db5b8231559a5cfc9 P 1d22290598ca4676a43fd50dc9cfc77e: Deleting consensus metadata
I20260924 19:31:58.137115 7774 ts_tablet_manager.cc:1918] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:58.137187 7774 log.cc:1196] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/afa79a8c8b1f441db5b8231559a5cfc9
I20260924 19:31:58.137414 7774 ts_tablet_manager.cc:1939] T afa79a8c8b1f441db5b8231559a5cfc9 P 30a1e8e885b14651aac5c8f8d58ac786: Deleting consensus metadata
W20260924 19:31:58.139758 7611 connection.cc:535] server connection from 127.0.0.1:58954 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 19:31:58.139761 7338 connection.cc:535] server connection from 127.0.0.1:58472 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:31:58.141481 7776 tablet_replica.cc:330] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: stopping tablet replica
I20260924 19:31:58.141597 7776 raft_consensus.cc:2229] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:31:58.141723 7776 raft_consensus.cc:2258] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:31:58.144016 7776 ts_tablet_manager.cc:1905] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:31:58.148324 7776 ts_tablet_manager.cc:1918] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:31:58.148411 7776 log.cc:1196] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/afa79a8c8b1f441db5b8231559a5cfc9
I20260924 19:31:58.148727 7776 ts_tablet_manager.cc:1939] T afa79a8c8b1f441db5b8231559a5cfc9 P a77223cab72e427ba7553a4480fe9d93: Deleting consensus metadata
W20260924 19:31:58.149123 7476 connection.cc:441] server connection from 127.0.0.1:54814 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54814 (request call id 1) could send its response
I20260924 19:31:58.150328 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.141486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.0.98.61:45165)
0924 19:31:58.141531 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.141533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.141653 (+ 120us) server_negotiation.cc:407] Connection header received
0924 19:31:58.141770 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.141771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.141781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.141855 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.142506 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.143275 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.144451 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.144777 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.144818 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.146451 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.146452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.146452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.146459 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.148424 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.148972 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.148973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.148974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.149029 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.149275 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.149276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.149276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.149401 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.149470 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.149928 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.150008 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:31:58.150956 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.141419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.98.60:39855)
0924 19:31:58.141456 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.141458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.141838 (+ 380us) server_negotiation.cc:407] Connection header received
0924 19:31:58.142565 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.142566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.142574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.142598 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.143263 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.143887 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.144903 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.145168 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.145210 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.146728 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.146730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.146730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.146737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.148238 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.148702 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.148703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.148703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.148769 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.148999 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.148999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.149000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.150288 (+ 1288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.150404 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.150689 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.150864 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:31:58.151988 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:58.132677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58314 (local address 127.0.98.60:39855)
0924 19:31:58.132806 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:58.132808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:58.140436 (+ 7628us) server_negotiation.cc:407] Connection header received
0924 19:31:58.140531 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:58.140533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:58.140543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:58.140570 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:31:58.141872 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.142625 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.145022 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:58.145849 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:58.145896 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:58.149276 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:31:58.149278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:31:58.149278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:31:58.149285 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:31:58.150915 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.151285 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.151285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.151286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.151344 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:31:58.151542 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:31:58.151542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:31:58.151543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:31:58.151658 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:31:58.151746 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:58.151840 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 19:31:58.151909 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 19:31:59.130097 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.129764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:47639)
0924 19:31:59.129943 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.130040 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
I20260924 19:31:59.130470 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.130069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:51127)
0924 19:31:59.130171 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.130425 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":47,"threads_started":1}
W20260924 19:31:59.130546 7451 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:31:59.130754 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.130525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:40407)
0924 19:31:59.130650 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.130715 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
W20260924 19:31:59.130779 7588 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
W20260924 19:31:59.130999 7724 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43727 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:31:59.632031 7787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b5cdc2713da469c9dcb55b7726d86ba)
I20260924 19:31:59.632156 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b5cdc2713da469c9dcb55b7726d86ba)
I20260924 19:31:59.632182 7787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } }
I20260924 19:31:59.632282 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } }
I20260924 19:31:59.632452 7787 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727), 251bd89115e241ba9883c87e0db592fa (127.0.98.60:39855)
I20260924 19:31:59.632695 7788 leader_election.cc:288] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727), 6fb75f62318049e8b2d64c65559219d2 (127.0.98.61:45165)
I20260924 19:31:59.632786 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.632581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41320)
0924 19:31:59.632698 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.632749 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
W20260924 19:31:59.632927 7258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:31:59.632958 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.632552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41308)
0924 19:31:59.632739 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.632922 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":51,"threads_started":1}
W20260924 19:31:59.633111 7070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:31:59.633474 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.633249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41324)
0924 19:31:59.633393 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.633445 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260924 19:31:59.633487 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.633412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41326)
0924 19:31:59.633443 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.633464 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 19:31:59.633546 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
W20260924 19:31:59.633579 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:31:59.635550 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.632855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45165 (local address 127.0.0.1:56480)
0924 19:31:59.632883 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.632891 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:59.632945 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:59.633227 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:59.633227 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:59.633231 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:59.633286 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:59.633288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:59.634131 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:59.634131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:59.634835 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:59.634836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:59.635331 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:59.635332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:59.635388 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:59.635390 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:31:59.635431 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:31:59.635473 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":1}
I20260924 19:31:59.635551 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.632725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.0.98.60:39855)
0924 19:31:59.633202 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:59.633204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:59.633211 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:31:59.633231 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:59.633232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:59.633241 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:59.633432 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:59.633463 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:59.634035 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:59.634929 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:59.635257 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:59.635314 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:59.635325 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:59.635454 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 19:31:59.635489 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":460,"thread_start_us":37,"threads_started":1}
I20260924 19:31:59.635558 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.632931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.0.98.61:45165)
0924 19:31:59.633016 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 19:31:59.633018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:31:59.633026 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:31:59.633051 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:31:59.633051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:31:59.633062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:31:59.633133 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:31:59.633365 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:59.634065 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:59.634927 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:31:59.635264 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:31:59.635321 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:59.635330 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:31:59.635446 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 19:31:59.635480 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":49,"threads_started":1}
I20260924 19:31:59.635568 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.632581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39855 (local address 127.0.0.1:58326)
0924 19:31:59.632803 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.632813 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:31:59.633081 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:31:59.633326 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:31:59.633327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:31:59.633329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:31:59.633374 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:59.633376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:59.634110 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:59.634111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:59.634845 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:31:59.634847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:31:59.635325 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:31:59.635326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:31:59.635383 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:31:59.635386 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:31:59.635441 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:31:59.635480 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":40,"threads_started":1}
I20260924 19:31:59.635848 7093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "251bd89115e241ba9883c87e0db592fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb75f62318049e8b2d64c65559219d2" is_pre_election: true
I20260924 19:31:59.635959 7093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 251bd89115e241ba9883c87e0db592fa in term 1.
I20260924 19:31:59.635933 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fb75f62318049e8b2d64c65559219d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "251bd89115e241ba9883c87e0db592fa" is_pre_election: true
I20260924 19:31:59.636019 7276 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6fb75f62318049e8b2d64c65559219d2 in term 1.
I20260924 19:31:59.636111 7257 leader_election.cc:302] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 251bd89115e241ba9883c87e0db592fa, 6fb75f62318049e8b2d64c65559219d2; no voters: 8b5cdc2713da469c9dcb55b7726d86ba
I20260924 19:31:59.636196 7788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:31:59.636266 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b5cdc2713da469c9dcb55b7726d86ba)
I20260924 19:31:59.636258 7069 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 251bd89115e241ba9883c87e0db592fa, 6fb75f62318049e8b2d64c65559219d2; no voters: 8b5cdc2713da469c9dcb55b7726d86ba
I20260924 19:31:59.636289 7788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:31:59.636364 7787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:31:59.636408 7787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b5cdc2713da469c9dcb55b7726d86ba)
I20260924 19:31:59.636427 7787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:31:59.637065 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } }
I20260924 19:31:59.637076 7787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } }
I20260924 19:31:59.637176 7788 leader_election.cc:288] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 2 election: Requested vote from peers 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727), 6fb75f62318049e8b2d64c65559219d2 (127.0.98.61:45165)
I20260924 19:31:59.637187 7787 leader_election.cc:288] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [CANDIDATE]: Term 2 election: Requested vote from peers 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727), 251bd89115e241ba9883c87e0db592fa (127.0.98.60:39855)
I20260924 19:31:59.637347 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.637270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41340)
0924 19:31:59.637300 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.637326 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 19:31:59.637327 7093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "251bd89115e241ba9883c87e0db592fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb75f62318049e8b2d64c65559219d2"
I20260924 19:31:59.637400 7093 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 251bd89115e241ba9883c87e0db592fa in current term 2: Already voted for candidate 6fb75f62318049e8b2d64c65559219d2 in this term.
I20260924 19:31:59.637476 7276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fb75f62318049e8b2d64c65559219d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "251bd89115e241ba9883c87e0db592fa"
I20260924 19:31:59.637548 7276 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6fb75f62318049e8b2d64c65559219d2 in current term 2: Already voted for candidate 251bd89115e241ba9883c87e0db592fa in this term.
I20260924 19:31:59.637661 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.637299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41350)
0924 19:31:59.637620 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.637641 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307}
I20260924 19:31:59.637661 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.637565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41356)
0924 19:31:59.637613 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.637641 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 19:31:59.637755 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:31:59.637801 7258 leader_election.cc:302] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 251bd89115e241ba9883c87e0db592fa; no voters: 6fb75f62318049e8b2d64c65559219d2, 8b5cdc2713da469c9dcb55b7726d86ba
I20260924 19:31:59.637887 7788 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 19:31:59.637928 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:31:59.637840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41372)
0924 19:31:59.637878 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:31:59.637907 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:31:59.638001 7070 leader_election.cc:334] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:31:59.638046 7070 leader_election.cc:302] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6fb75f62318049e8b2d64c65559219d2; no voters: 251bd89115e241ba9883c87e0db592fa, 8b5cdc2713da469c9dcb55b7726d86ba
I20260924 19:31:59.638125 7787 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 19:32:00.131376 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:00.131171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:50093)
0924 19:32:00.131281 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:00.131321 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
I20260924 19:32:00.131429 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:00.130969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:39601)
0924 19:32:00.131341 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:00.131380 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260924 19:32:00.131474 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:00.131264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:35605)
0924 19:32:00.131409 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:00.131437 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":44,"threads_started":1}
I20260924 19:32:01.132220 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.132020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:40131)
0924 19:32:01.132134 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.132169 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260924 19:32:01.132170 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.131934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:39811)
0924 19:32:01.132069 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.132111 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260924 19:32:01.132525 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.131934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:34033)
0924 19:32:01.132448 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.132484 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":60,"threads_started":1}
I20260924 19:32:01.572785 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:32:01.572885 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } }
I20260924 19:32:01.573033 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727), 6fb75f62318049e8b2d64c65559219d2 (127.0.98.61:45165)
I20260924 19:32:01.573298 7093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "251bd89115e241ba9883c87e0db592fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb75f62318049e8b2d64c65559219d2" is_pre_election: true
I20260924 19:32:01.573377 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.573190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41606)
0924 19:32:01.573297 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.573336 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260924 19:32:01.573393 7093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 251bd89115e241ba9883c87e0db592fa in term 2.
I20260924 19:32:01.573581 7257 leader_election.cc:302] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 251bd89115e241ba9883c87e0db592fa, 6fb75f62318049e8b2d64c65559219d2; no voters:
I20260924 19:32:01.573694 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 19:32:01.573743 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:32:01.573768 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:32:01.573855 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.573764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41616)
0924 19:32:01.573806 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.573832 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:32:01.573926 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:32:01.574505 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } }
I20260924 19:32:01.574618 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 3 election: Requested vote from peers 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727), 6fb75f62318049e8b2d64c65559219d2 (127.0.98.61:45165)
I20260924 19:32:01.574844 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.574732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41630)
0924 19:32:01.574789 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.574820 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 19:32:01.574810 7093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "251bd89115e241ba9883c87e0db592fa" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6fb75f62318049e8b2d64c65559219d2"
I20260924 19:32:01.574889 7093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:32:01.575151 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.575064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:41642)
0924 19:32:01.575106 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.575131 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 19:32:01.575246 7258 leader_election.cc:334] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
I20260924 19:32:01.575570 7093 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 251bd89115e241ba9883c87e0db592fa in term 3.
I20260924 19:32:01.575729 7257 leader_election.cc:302] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 251bd89115e241ba9883c87e0db592fa, 6fb75f62318049e8b2d64c65559219d2; no voters: 8b5cdc2713da469c9dcb55b7726d86ba
I20260924 19:32:01.575822 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:32:01.575930 7803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [term 3 LEADER]: Becoming Leader. State: Replica: 251bd89115e241ba9883c87e0db592fa, State: Running, Role: LEADER
I20260924 19:32:01.576010 7803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } }
I20260924 19:32:01.576356 7806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 251bd89115e241ba9883c87e0db592fa. Latest consensus state: current_term: 3 leader_uuid: "251bd89115e241ba9883c87e0db592fa" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:32:01.576458 7806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:01.576686 7808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:32:01.577105 7808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:32:01.577162 7808 catalog_manager.cc:1259] Loaded cluster ID: af5bc9eb2aad45d0a00044e1d10d36dc
I20260924 19:32:01.577178 7808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:32:01.577339 7808 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:32:01.577420 7808 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Loaded TSK: 0
I20260924 19:32:01.577551 7808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:32:01.636104 7270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:32:01.637689 7270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:32:01.638620 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.638517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:35500)
0924 19:32:01.638566 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.638592 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:32:01.638644 7093 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [term 3 FOLLOWER]: Refusing update from remote peer 251bd89115e241ba9883c87e0db592fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 19:32:01.638859 7806 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 19:32:01.638936 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.638851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.0.1:35504)
0924 19:32:01.638894 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.638919 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 19:32:01.639027 7258 consensus_peers.cc:487] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa -> Peer 8b5cdc2713da469c9dcb55b7726d86ba (127.0.98.62:43727): Couldn't send request to peer 8b5cdc2713da469c9dcb55b7726d86ba. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:32:01.639406 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 251bd89115e241ba9883c87e0db592fa. Latest consensus state: current_term: 3 leader_uuid: "251bd89115e241ba9883c87e0db592fa" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:32:01.639501 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:32:01.639798 7809 mvcc.cc:205] Tried to move back new op lower bound from 7332980005429915648 to 7332980005175885824. Current Snapshot: MvccSnapshot[applied={T|T < 7332980005429915648}]
I20260924 19:32:01.639904 7806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "251bd89115e241ba9883c87e0db592fa" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:32:01.640008 7806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:01.640015 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "251bd89115e241ba9883c87e0db592fa" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8b5cdc2713da469c9dcb55b7726d86ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43727 } } peers { permanent_uuid: "6fb75f62318049e8b2d64c65559219d2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45165 } attrs { promote: false } } peers { permanent_uuid: "251bd89115e241ba9883c87e0db592fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39855 } attrs { promote: false } } }
I20260924 19:32:01.640053 7811 mvcc.cc:205] Tried to move back new op lower bound from 7332980005429915648 to 7332980005175885824. Current Snapshot: MvccSnapshot[applied={T|T < 7332980005429915648}]
I20260924 19:32:01.640080 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fb75f62318049e8b2d64c65559219d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:32:01.645097 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.641959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.98.1:38451)
0924 19:32:01.642085 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:01.642087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:01.642097 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:01.642123 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:01.642123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:01.642134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:01.642224 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:01.642391 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.643028 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.644150 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.644787 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.644846 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.644862 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:01.644985 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:32:01.645022 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260924 19:32:01.645092 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.641875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:48354)
0924 19:32:01.641916 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.641927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:01.641978 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:01.642247 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:01.642248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:01.642249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:01.642316 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.642317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.643094 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.643094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.644057 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.644059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.644851 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.644852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.644912 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.644917 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:32:01.644973 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:32:01.645016 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 19:32:01.645398 7384 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbebda984234f97a4b4047c32b2af39 (DEFAULT_TABLE table=test-table [id=9bb6ad7fe85a4fdfb68a3d6c4a244228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:01.645525 7384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cbebda984234f97a4b4047c32b2af39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:01.645521 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.641964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.0.1:48472)
0924 19:32:01.642476 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.642483 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:01.642549 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:01.642702 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:01.642703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:01.642705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:01.642740 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.642741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.643576 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.643577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.644267 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.644268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.645140 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.645141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.645361 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.645367 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:32:01.645410 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:32:01.645452 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":56,"threads_started":1}
I20260924 19:32:01.645620 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.641885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41535 (local address 127.0.0.1:41014)
0924 19:32:01.642084 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.642094 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:01.642133 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:01.642350 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:01.642351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:01.642353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:01.642400 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.642402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.643476 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.643477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.644499 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.644500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.645429 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.645430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.645490 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.645493 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:01.645533 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 19:32:01.645566 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260924 19:32:01.645748 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.642098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.0.98.2:39351)
0924 19:32:01.642193 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:01.642195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:01.642525 (+ 330us) server_negotiation.cc:407] Connection header received
0924 19:32:01.642624 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:01.642625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:01.642637 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:01.642688 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:01.642792 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.643502 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.644403 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.645032 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.645095 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.645108 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:01.645643 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 19:32:01.645688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":48,"threads_started":1}
I20260924 19:32:01.645938 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.642108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.0.98.3:41535)
0924 19:32:01.642247 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:01.642249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:01.642254 (+ 5us) server_negotiation.cc:407] Connection header received
0924 19:32:01.642275 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:01.642276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:01.642283 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:01.642335 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:01.642665 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.643398 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.644563 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.645125 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.645383 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.645401 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:01.645725 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 19:32:01.645776 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":50,"threads_started":1}
I20260924 19:32:01.645958 7521 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbebda984234f97a4b4047c32b2af39 (DEFAULT_TABLE table=test-table [id=9bb6ad7fe85a4fdfb68a3d6c4a244228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:01.646060 7521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cbebda984234f97a4b4047c32b2af39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:01.646867 7821 tablet_bootstrap.cc:492] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786: Bootstrap starting.
I20260924 19:32:01.647204 7821 tablet_bootstrap.cc:654] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:01.647353 7822 tablet_bootstrap.cc:492] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: Bootstrap starting.
I20260924 19:32:01.647459 7657 tablet_service.cc:1467] Processing CreateTablet for tablet 2cbebda984234f97a4b4047c32b2af39 (DEFAULT_TABLE table=test-table [id=9bb6ad7fe85a4fdfb68a3d6c4a244228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:01.647565 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cbebda984234f97a4b4047c32b2af39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:01.647692 7821 tablet_bootstrap.cc:492] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786: No bootstrap required, opened a new log
I20260924 19:32:01.647725 7821 ts_tablet_manager.cc:1397] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:01.647737 7822 tablet_bootstrap.cc:654] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:01.647818 7821 raft_consensus.cc:348] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.647868 7821 raft_consensus.cc:374] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:01.647881 7821 raft_consensus.cc:729] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a1e8e885b14651aac5c8f8d58ac786, State: Initialized, Role: FOLLOWER
I20260924 19:32:01.647933 7821 consensus_queue.cc:260] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.648077 7821 ts_tablet_manager.cc:1428] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:01.648356 7822 tablet_bootstrap.cc:492] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: No bootstrap required, opened a new log
I20260924 19:32:01.648397 7822 ts_tablet_manager.cc:1397] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:01.648500 7822 raft_consensus.cc:348] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.648552 7822 raft_consensus.cc:374] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:01.648568 7822 raft_consensus.cc:729] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77223cab72e427ba7553a4480fe9d93, State: Initialized, Role: FOLLOWER
I20260924 19:32:01.648600 7822 consensus_queue.cc:260] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.648751 7822 ts_tablet_manager.cc:1428] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:01.648886 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.648329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:59169)
0924 19:32:01.648803 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.648841 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":72,"threads_started":1}
I20260924 19:32:01.649045 7450 heartbeater.cc:503] Master 127.0.98.60:39855 was elected leader, sending a full tablet report...
I20260924 19:32:01.649557 7587 heartbeater.cc:503] Master 127.0.98.60:39855 was elected leader, sending a full tablet report...
I20260924 19:32:01.649524 7828 tablet_bootstrap.cc:492] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: Bootstrap starting.
I20260924 19:32:01.649902 7828 tablet_bootstrap.cc:654] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:01.650156 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.649509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:37965)
0924 19:32:01.649986 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.650131 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458}
I20260924 19:32:01.650363 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.649203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:60067)
0924 19:32:01.650292 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.650326 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":74,"threads_started":1}
I20260924 19:32:01.650476 7828 tablet_bootstrap.cc:492] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: No bootstrap required, opened a new log
I20260924 19:32:01.650523 7828 ts_tablet_manager.cc:1397] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:01.650640 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.650559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:50087)
0924 19:32:01.650590 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.650614 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 19:32:01.650625 7828 raft_consensus.cc:348] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.650672 7828 raft_consensus.cc:374] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:01.650686 7828 raft_consensus.cc:729] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d22290598ca4676a43fd50dc9cfc77e, State: Initialized, Role: FOLLOWER
I20260924 19:32:01.650724 7828 consensus_queue.cc:260] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.650849 7828 ts_tablet_manager.cc:1428] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:01.651250 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.650990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:51577)
0924 19:32:01.651185 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.651218 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260924 19:32:01.651263 7723 heartbeater.cc:503] Master 127.0.98.60:39855 was elected leader, sending a full tablet report...
I20260924 19:32:01.651520 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.651438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:41955)
0924 19:32:01.651472 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.651499 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 19:32:01.660908 7590 tablet.cc:2369] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:32:01.779745 7827 raft_consensus.cc:482] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:32:01.779856 7827 raft_consensus.cc:504] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.780261 7827 leader_election.cc:288] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451), 1d22290598ca4676a43fd50dc9cfc77e (127.0.98.3:41535)
I20260924 19:32:01.784070 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.780440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41535 (local address 127.0.98.2:43099)
0924 19:32:01.780497 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.780509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:01.780566 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:01.780745 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:01.780745 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:01.780748 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:01.780810 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.780812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.781725 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.781727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.783014 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.783018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.783798 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.783799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.783883 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.783888 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:32:01.783955 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:32:01.784001 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2}
I20260924 19:32:01.784083 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.780553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43099 (local address 127.0.98.3:41535)
0924 19:32:01.780592 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:01.780594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:01.780602 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:32:01.780646 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:01.780647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:01.780659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:01.780716 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:01.780883 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.781646 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.783124 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.783715 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.783784 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.783796 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:01.783973 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 19:32:01.784018 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:32:01.784094 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.780521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40901 (local address 127.0.98.1:38451)
0924 19:32:01.780563 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:01.780565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:01.780758 (+ 193us) server_negotiation.cc:407] Connection header received
0924 19:32:01.780805 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:01.780806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:01.780844 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:01.780898 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:01.781011 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.781682 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.783125 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.783709 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.783789 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.783800 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:01.784005 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:32:01.784042 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:32:01.784296 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.780441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.98.2:40901)
0924 19:32:01.780695 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.780704 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:01.780754 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:01.780907 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:01.780908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:01.780909 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:01.780956 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.780958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.781746 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.781746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.783014 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:01.783018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.783865 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.783866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:01.783923 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.783927 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:32:01.784161 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 19:32:01.784206 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":151,"thread_start_us":134,"threads_started":1}
I20260924 19:32:01.784401 7677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbebda984234f97a4b4047c32b2af39" candidate_uuid: "a77223cab72e427ba7553a4480fe9d93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d22290598ca4676a43fd50dc9cfc77e" is_pre_election: true
I20260924 19:32:01.784531 7677 raft_consensus.cc:2454] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a77223cab72e427ba7553a4480fe9d93 in term 0.
I20260924 19:32:01.784653 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbebda984234f97a4b4047c32b2af39" candidate_uuid: "a77223cab72e427ba7553a4480fe9d93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a1e8e885b14651aac5c8f8d58ac786" is_pre_election: true
I20260924 19:32:01.784727 7404 raft_consensus.cc:2454] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a77223cab72e427ba7553a4480fe9d93 in term 0.
I20260924 19:32:01.784736 7476 leader_election.cc:302] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d22290598ca4676a43fd50dc9cfc77e, a77223cab72e427ba7553a4480fe9d93; no voters:
I20260924 19:32:01.784968 7827 raft_consensus.cc:2781] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:32:01.785009 7827 raft_consensus.cc:482] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:32:01.785024 7827 raft_consensus.cc:3037] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:01.785617 7827 raft_consensus.cc:504] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.785876 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbebda984234f97a4b4047c32b2af39" candidate_uuid: "a77223cab72e427ba7553a4480fe9d93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30a1e8e885b14651aac5c8f8d58ac786"
I20260924 19:32:01.785889 7677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cbebda984234f97a4b4047c32b2af39" candidate_uuid: "a77223cab72e427ba7553a4480fe9d93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d22290598ca4676a43fd50dc9cfc77e"
I20260924 19:32:01.785951 7677 raft_consensus.cc:3037] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:01.785954 7404 raft_consensus.cc:3037] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:32:01.786329 7453 tablet.cc:2369] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:32:01.786636 7677 raft_consensus.cc:2454] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a77223cab72e427ba7553a4480fe9d93 in term 1.
I20260924 19:32:01.786655 7404 raft_consensus.cc:2454] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a77223cab72e427ba7553a4480fe9d93 in term 1.
I20260924 19:32:01.785835 7827 leader_election.cc:288] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [CANDIDATE]: Term 1 election: Requested vote from peers 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451), 1d22290598ca4676a43fd50dc9cfc77e (127.0.98.3:41535)
I20260924 19:32:01.786801 7476 leader_election.cc:302] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d22290598ca4676a43fd50dc9cfc77e, a77223cab72e427ba7553a4480fe9d93; no voters:
I20260924 19:32:01.786928 7827 raft_consensus.cc:2781] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:01.787088 7827 raft_consensus.cc:686] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 1 LEADER]: Becoming Leader. State: Replica: a77223cab72e427ba7553a4480fe9d93, State: Running, Role: LEADER
I20260924 19:32:01.787153 7827 consensus_queue.cc:237] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } }
I20260924 19:32:01.787425 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.787321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:43179)
0924 19:32:01.787369 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.787399 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:32:01.787842 7271 catalog_manager.cc:5495] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 reported cstate change: term changed from 0 to 1, leader changed from <none> to a77223cab72e427ba7553a4480fe9d93 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a77223cab72e427ba7553a4480fe9d93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a77223cab72e427ba7553a4480fe9d93" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 } health_report { overall_health: UNKNOWN } } }
W20260924 19:32:01.793282 7726 tablet.cc:2369] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:32:01.826927 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.824616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.0.98.2:39351)
0924 19:32:01.824689 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:01.824690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:01.824700 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:01.824801 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:01.824802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:01.824827 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:01.824891 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:01.825032 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.825515 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.826232 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:01.826425 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:01.826479 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:01.826622 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:01.826721 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:01.826781 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:01.826824 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:32:01.826848 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 19:32:01.827960 7404 raft_consensus.cc:1261] T 2cbebda984234f97a4b4047c32b2af39 P 30a1e8e885b14651aac5c8f8d58ac786 [term 1 FOLLOWER]: Refusing update from remote peer a77223cab72e427ba7553a4480fe9d93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:01.828106 7677 raft_consensus.cc:1261] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 FOLLOWER]: Refusing update from remote peer a77223cab72e427ba7553a4480fe9d93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:01.828239 7834 consensus_queue.cc:1035] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30a1e8e885b14651aac5c8f8d58ac786" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:01.828378 7827 consensus_queue.cc:1035] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d22290598ca4676a43fd50dc9cfc77e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:01.829567 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.829313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:52349)
0924 19:32:01.829499 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.829534 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260924 19:32:01.829713 7837 mvcc.cc:205] Tried to move back new op lower bound from 7332980006205366272 to 7332980006040645632. Current Snapshot: MvccSnapshot[applied={T|T < 7332980006205366272}]
I20260924 19:32:01.829857 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.829712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:56929)
0924 19:32:01.829750 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.829825 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:32:01.830063 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.829963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:51717)
0924 19:32:01.830007 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.830037 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:32:01.830411 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.830325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.1:56863)
0924 19:32:01.830362 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.830385 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 19:32:01.831317 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.831013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.2:49531)
0924 19:32:01.831258 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.831293 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228}
I20260924 19:32:01.831761 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:01.831607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43727 (local address 127.0.98.3:55007)
0924 19:32:01.831704 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.831734 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 19:32:01.832422 7271 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:01.832506 7271 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:01.833657 7271 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 251bd89115e241ba9883c87e0db592fa: Sending DeleteTablet for 3 replicas of tablet 2cbebda984234f97a4b4047c32b2af39
I20260924 19:32:01.833941 7657 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cbebda984234f97a4b4047c32b2af39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:01 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41014
I20260924 19:32:01.834017 7384 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cbebda984234f97a4b4047c32b2af39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:01 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48354
I20260924 19:32:01.834172 7521 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cbebda984234f97a4b4047c32b2af39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:01 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48472
I20260924 19:32:01.834280 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 7231
I20260924 19:32:01.835614 7846 tablet_replica.cc:330] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: stopping tablet replica
I20260924 19:32:01.835672 7846 raft_consensus.cc:2229] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:01.835747 7846 raft_consensus.cc:2258] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:01.836052 7846 ts_tablet_manager.cc:1905] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:01.836733 7848 tablet_replica.cc:330] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: stopping tablet replica
I20260924 19:32:01.836849 7848 raft_consensus.cc:2229] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:32:01.836982 7848 raft_consensus.cc:2258] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:01.837280 7848 ts_tablet_manager.cc:1905] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:01.837598 7846 ts_tablet_manager.cc:1918] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:01.837694 7846 log.cc:1196] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/2cbebda984234f97a4b4047c32b2af39
I20260924 19:32:01.837986 7846 ts_tablet_manager.cc:1939] T 2cbebda984234f97a4b4047c32b2af39 P 1d22290598ca4676a43fd50dc9cfc77e: Deleting consensus metadata
I20260924 19:32:01.838351 7257 catalog_manager.cc:4841] TS 1d22290598ca4676a43fd50dc9cfc77e (127.0.98.3:41535): tablet 2cbebda984234f97a4b4047c32b2af39 (table test-table [id=9bb6ad7fe85a4fdfb68a3d6c4a244228]) successfully deleted
I20260924 19:32:01.838825 7848 ts_tablet_manager.cc:1918] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:01.838872 7848 log.cc:1196] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/2cbebda984234f97a4b4047c32b2af39
I20260924 19:32:01.839056 7848 ts_tablet_manager.cc:1939] T 2cbebda984234f97a4b4047c32b2af39 P a77223cab72e427ba7553a4480fe9d93: Deleting consensus metadata
I20260924 19:32:01.839510 7255 catalog_manager.cc:4841] TS a77223cab72e427ba7553a4480fe9d93 (127.0.98.2:39351): tablet 2cbebda984234f97a4b4047c32b2af39 (table test-table [id=9bb6ad7fe85a4fdfb68a3d6c4a244228]) successfully deleted
W20260924 19:32:01.840004 7817 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:01.839815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:48370)
0924 19:32:01.839866 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:01.839877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:01.839929 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:01.839967 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38451: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:32:01.840157 7255 catalog_manager.cc:4584] TS 30a1e8e885b14651aac5c8f8d58ac786 (127.0.98.1:38451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cbebda984234f97a4b4047c32b2af39: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38451: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260924 19:32:01.840327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 7458
I20260924 19:32:01.845460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 7594
I20260924 19:32:01.850860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 6948
I20260924 19:32:01.854533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 7137
2026-09-24T19:32:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6354 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:32:01 dist-test-slave-m8jb krb5kdc[7851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:32:01 dist-test-slave-m8jb krb5kdc[7851](info): set up 2 sockets
Sep 24 19:32:01 dist-test-slave-m8jb krb5kdc[7851](info): commencing operation
krb5kdc: starting...
W20260924 19:32:02.824894 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:32:02 dist-test-slave-m8jb krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:32:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 19:32:02.943742 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:03.022842 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:03.023067 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:03.023094 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:03.024266 7864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:32:03.024307 7864 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:32:03.024324 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:03.024341 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:03.024356 7864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:32:03.024370 7864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:32:03.025825 7864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42613
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:03.026074 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:03.026275 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:03.031031 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:03.031046 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:03.031180 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:03.031484 7864 server_base.cc:1034] running on GCE node
I20260924 19:32:03.031706 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:03.031950 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:03.033107 7864 hybrid_clock.cc:648] HybridClock initialized: now 1790278323033091 us; error 29 us; skew 500 ppm
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:03.035477 7864 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 19:32:03.035876 7864 webserver.cc:460] Webserver started at http://127.0.98.62:37659/ using document root <none> and password file <none>
I20260924 19:32:03.036103 7864 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:03.036150 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:03.036275 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:03.037343 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "59ff2a98d540461a95b8be58e6574c62"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "0d9643d8f7d21de6b0f2da1d4b9b2ab5"
server_key_iv: "9f46bc900419a28134caba7c20f34023"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.037676 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "59ff2a98d540461a95b8be58e6574c62"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "0d9643d8f7d21de6b0f2da1d4b9b2ab5"
server_key_iv: "9f46bc900419a28134caba7c20f34023"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.038890 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:32:03.039574 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.039722 7864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:03.039803 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "59ff2a98d540461a95b8be58e6574c62"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "0d9643d8f7d21de6b0f2da1d4b9b2ab5"
server_key_iv: "9f46bc900419a28134caba7c20f34023"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.039861 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:03.051718 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:03.053504 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:03.053634 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:03.057857 7864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42613
I20260924 19:32:03.057915 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42613 every 8 connection(s)
I20260924 19:32:03.058215 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:32:03.058810 7942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.061017 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: Bootstrap starting.
I20260924 19:32:03.061520 7942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.061745 7942 log.cc:824] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:03.062394 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: No bootstrap required, opened a new log
I20260924 19:32:03.064479 7942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff2a98d540461a95b8be58e6574c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42613 } }
I20260924 19:32:03.064625 7942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.064692 7942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ff2a98d540461a95b8be58e6574c62, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.064834 7942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff2a98d540461a95b8be58e6574c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42613 } }
I20260924 19:32:03.064913 7942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:32:03.064945 7942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:32:03.064980 7942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.065457 7942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff2a98d540461a95b8be58e6574c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42613 } }
I20260924 19:32:03.065546 7942 leader_election.cc:302] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59ff2a98d540461a95b8be58e6574c62; no voters:
I20260924 19:32:03.065673 7942 leader_election.cc:288] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:32:03.065727 7945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:03.065899 7945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [term 1 LEADER]: Becoming Leader. State: Replica: 59ff2a98d540461a95b8be58e6574c62, State: Running, Role: LEADER
I20260924 19:32:03.065915 7942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:32:03.066051 7945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff2a98d540461a95b8be58e6574c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42613 } }
I20260924 19:32:03.066393 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59ff2a98d540461a95b8be58e6574c62. Latest consensus state: current_term: 1 leader_uuid: "59ff2a98d540461a95b8be58e6574c62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff2a98d540461a95b8be58e6574c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42613 } } }
I20260924 19:32:03.066395 7946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59ff2a98d540461a95b8be58e6574c62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59ff2a98d540461a95b8be58e6574c62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42613 } } }
I20260924 19:32:03.066645 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:03.066653 7946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:03.066992 7954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:32:03.067480 7954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:32:03.068125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 7864
I20260924 19:32:03.068241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:32:03.068418 392 external_mini_cluster.cc:1428] Setting key 27bc69f2ddf837cc9ad8f03761b1009f
I20260924 19:32:03.069161 7954 catalog_manager.cc:1347] Generated new cluster ID: 071b4f5ad58e4b12a9c34b718f353415
I20260924 19:32:03.069212 7954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:32:03.079272 7954 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:32:03.079969 7954 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:32:03.080122 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.069008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.0.98.62:42613)
0924 19:32:03.069148 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.069157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.069189 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:32:03.069302 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.069310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.069366 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.069527 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.070225 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.071666 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.072703 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.073366 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.073500 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.076150 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.076155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.076156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.076167 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.078097 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.078498 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.078501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.078502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.078627 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.078839 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.078840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.078840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.079125 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.079253 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.079686 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.079829 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260924 19:32:03.087102 7954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: Generated new TSK 0
I20260924 19:32:03.087314 7954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 19:32:03.123448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42613
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:03.212042 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:03.212252 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:03.212276 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:03.213451 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:03.213492 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:03.213512 7968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:32:03.213527 7968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:32:03.214958 7968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:03.215161 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:03.215381 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:03.220301 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:03.220336 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:03.220400 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:03.220551 7968 server_base.cc:1034] running on GCE node
I20260924 19:32:03.220781 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:03.221042 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:03.222182 7968 hybrid_clock.cc:648] HybridClock initialized: now 1790278323222168 us; error 34 us; skew 500 ppm
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:03.224514 7968 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 19:32:03.225001 7968 webserver.cc:460] Webserver started at http://127.0.98.1:33281/ using document root <none> and password file <none>
I20260924 19:32:03.225191 7968 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:03.225236 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:03.225351 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:03.226341 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "f53ba118edac41aa99dfbef3d554d695"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "3b84aafa420b8e88520bab34eb857ac1"
server_key_iv: "12b9e2ed73f4eb87690c859dd725003b"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.226650 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f53ba118edac41aa99dfbef3d554d695"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "3b84aafa420b8e88520bab34eb857ac1"
server_key_iv: "12b9e2ed73f4eb87690c859dd725003b"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.227828 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:32:03.228438 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.228585 7968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:03.228682 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "f53ba118edac41aa99dfbef3d554d695"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "3b84aafa420b8e88520bab34eb857ac1"
server_key_iv: "12b9e2ed73f4eb87690c859dd725003b"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.228742 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:03.243383 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:03.245414 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:03.245524 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:03.246018 7968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:03.246058 7968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.246081 7968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:03.246093 7968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:32:03.252521 7968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36321
I20260924 19:32:03.252588 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36321 every 8 connection(s)
I20260924 19:32:03.252925 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:32:03.255008 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.246445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44083 (local address 127.0.98.62:42613)
0924 19:32:03.246512 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.246515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.246669 (+ 154us) server_negotiation.cc:407] Connection header received
0924 19:32:03.247164 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.247165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.247181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.247253 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.247894 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.248707 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.249703 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.249919 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.249976 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.252575 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.252576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.252576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.252588 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.253911 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.254275 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.254276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.254277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.254329 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.254511 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.254511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.254511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.254607 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.254649 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.254875 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.254947 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:32:03.255146 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.246406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42613 (local address 127.0.98.1:44083)
0924 19:32:03.246546 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.246586 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.247071 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.247326 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.247331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.247598 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:03.247802 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.247807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.248795 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.248796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.249617 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.249619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.250003 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.250004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.250115 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.250583 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:03.250592 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:03.252463 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:03.253984 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.253989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.253991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.254201 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.254380 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.254380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.254380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.254468 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.254640 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:03.254643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:03.254749 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.254851 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.254968 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260924 19:32:03.255586 8099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42613
I20260924 19:32:03.255661 8099 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:03.255822 8099 heartbeater.cc:511] Master 127.0.98.62:42613 requested a full tablet report, sending...
I20260924 19:32:03.256345 7895 ts_manager.cc:194] Registered new tserver with Master: f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321)
I20260924 19:32:03.257079 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44083
I20260924 19:32:03.258502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 7968
I20260924 19:32:03.258595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:32:03.258781 392 external_mini_cluster.cc:1428] Setting key 11ae80d06821a4a27821811ec1af50eb
I20260924 19:32:03.262208 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.256511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47551 (local address 127.0.98.62:42613)
0924 19:32:03.256548 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.256550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.256664 (+ 114us) server_negotiation.cc:407] Connection header received
0924 19:32:03.256731 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.256732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.256741 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.256798 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.257078 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.257617 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.258391 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.258639 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.258708 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.259750 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.259751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.259751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.259758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.261117 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.261586 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.261587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.261587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.261641 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.261819 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.261820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.261820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.261912 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.261958 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.262091 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.262152 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:32:03.262275 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.256474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42613 (local address 127.0.98.1:47551)
0924 19:32:03.256595 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.256604 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.256677 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.256802 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.256803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.256964 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:03.257015 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.257017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.257697 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.257698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.258226 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.258228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.258763 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.258764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.258818 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.259202 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:03.259205 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:03.259541 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:03.261203 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.261205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.261205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.261495 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.261705 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.261705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.261705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.261772 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.261959 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:03.261960 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:03.262003 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.262075 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.262178 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 19:32:03.283210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42613
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:03.363649 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:03.363876 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:03.363909 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:03.365087 8108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:03.365130 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:03.365159 8108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:32:03.365180 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:32:03.366670 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:03.366910 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:03.367117 8108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:03.371958 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:03.372103 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:03.371985 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:03.372177 8108 server_base.cc:1034] running on GCE node
I20260924 19:32:03.372364 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:03.372565 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:03.373690 8108 hybrid_clock.cc:648] HybridClock initialized: now 1790278323373679 us; error 32 us; skew 500 ppm
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:03.376015 8108 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 19:32:03.376451 8108 webserver.cc:460] Webserver started at http://127.0.98.2:44341/ using document root <none> and password file <none>
I20260924 19:32:03.376683 8108 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:03.376732 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:03.376853 8108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:03.377880 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "0dce50ac5140488a99bf57687e680b27"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "6cfad8335be775d34b67d0cd44cf7127"
server_key_iv: "d96352f9420b438050e010ba8789fc2f"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.378216 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0dce50ac5140488a99bf57687e680b27"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "6cfad8335be775d34b67d0cd44cf7127"
server_key_iv: "d96352f9420b438050e010ba8789fc2f"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.379410 8108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:03.380084 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.380221 8108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:03.380300 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "0dce50ac5140488a99bf57687e680b27"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "6cfad8335be775d34b67d0cd44cf7127"
server_key_iv: "d96352f9420b438050e010ba8789fc2f"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.380364 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:03.398644 8108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:03.400488 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:03.400621 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:03.401223 8108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:03.401270 8108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.401299 8108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:03.401321 8108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:32:03.408104 8108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32963
I20260924 19:32:03.408166 8238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32963 every 8 connection(s)
I20260924 19:32:03.408476 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:32:03.411010 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.401582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58387 (local address 127.0.98.62:42613)
0924 19:32:03.401648 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.401650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.401874 (+ 224us) server_negotiation.cc:407] Connection header received
0924 19:32:03.402547 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.402549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.402558 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.402603 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.403444 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.404193 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.405796 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.406016 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.406082 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.408366 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.408367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.408367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.408374 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.409789 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.410151 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.410153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.410153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.410210 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.410420 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.410421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.410421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.410532 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.410585 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.410838 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.410911 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:32:03.411165 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.401579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42613 (local address 127.0.98.2:58387)
0924 19:32:03.401727 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.401775 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.402429 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.402687 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.402692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.403046 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:03.403338 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.403345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.404284 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.404285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.405706 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.405710 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.406110 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.406111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.406230 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.406672 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:03.406680 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:03.408264 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:03.409864 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.409870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.409872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.410084 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.410261 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.410262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.410262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.410365 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.410584 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:03.410590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:03.410720 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.410806 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.410957 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260924 19:32:03.411594 8239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42613
I20260924 19:32:03.411675 8239 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:03.411837 8239 heartbeater.cc:511] Master 127.0.98.62:42613 requested a full tablet report, sending...
I20260924 19:32:03.412209 7896 ts_manager.cc:194] Registered new tserver with Master: 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963)
I20260924 19:32:03.412693 7896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58387
I20260924 19:32:03.417074 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.412523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47873 (local address 127.0.98.62:42613)
0924 19:32:03.412557 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.412559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.412652 (+ 93us) server_negotiation.cc:407] Connection header received
0924 19:32:03.412732 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.412733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.412740 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.412793 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.413097 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.413546 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.414204 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.414391 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.414444 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.415139 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.415140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.415140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.415146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.416089 (+ 943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.416361 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.416362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.416363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.416413 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.416563 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.416564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.416564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.416740 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.416796 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.416930 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.416992 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:32:03.417088 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.412479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42613 (local address 127.0.98.2:47873)
0924 19:32:03.412595 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.412604 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.412681 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.412818 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.412819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.412977 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:03.413032 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.413033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.413628 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.413629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.414139 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.414141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.414448 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.414449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.414511 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.414826 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:03.414829 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:03.415069 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:03.416160 (+ 1091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.416161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.416162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.416306 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.416464 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.416464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.416465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.416517 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.416794 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:03.416795 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:03.416836 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.416896 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.416987 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260924 19:32:03.417886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 8108
I20260924 19:32:03.417987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:32:03.418190 392 external_mini_cluster.cc:1428] Setting key 46d0f21971cd5ff9614dfae76ee55b0d
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 19:32:03.440977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42613
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:03.519479 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:03.519681 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:03.519712 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:03.520884 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:03.520922 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:03.520948 8248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:32:03.520979 8248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:32:03.522415 8248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:03.522645 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:03.522848 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:03.527578 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:03.527586 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:03.527601 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:03.527756 8248 server_base.cc:1034] running on GCE node
I20260924 19:32:03.528031 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:03.528236 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:03.529369 8248 hybrid_clock.cc:648] HybridClock initialized: now 1790278323529354 us; error 31 us; skew 500 ppm
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:03.531764 8248 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 19:32:03.532204 8248 webserver.cc:460] Webserver started at http://127.0.98.3:35019/ using document root <none> and password file <none>
I20260924 19:32:03.532397 8248 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:03.532442 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:03.532563 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:03.533595 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "6874673348a74969a155aa93342e167e"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "c8b44942077156b93e42dc33b6c40da2"
server_key_iv: "f0721fb156925089794a750880bed7b7"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.533919 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6874673348a74969a155aa93342e167e"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "c8b44942077156b93e42dc33b6c40da2"
server_key_iv: "f0721fb156925089794a750880bed7b7"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.535137 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:32:03.535820 8263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.535979 8248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:03.536058 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "6874673348a74969a155aa93342e167e"
format_stamp: "Formatted at 2026-09-24 19:32:03 on dist-test-slave-m8jb"
server_key: "c8b44942077156b93e42dc33b6c40da2"
server_key_iv: "f0721fb156925089794a750880bed7b7"
server_key_version: "encryptionkey@0"
I20260924 19:32:03.536123 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:03.574203 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:03.576337 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:03.576475 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:03.577005 8248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:03.577044 8248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.577073 8248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:03.577099 8248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:32:03.583472 8248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41567
I20260924 19:32:03.583539 8378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41567 every 8 connection(s)
I20260924 19:32:03.583853 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:32:03.585887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 8248
I20260924 19:32:03.586016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:32:03.586197 392 external_mini_cluster.cc:1428] Setting key e29e63682d5b7c931468f6199cee2788
I20260924 19:32:03.586316 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.577435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44991 (local address 127.0.98.62:42613)
0924 19:32:03.577478 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.577480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.577687 (+ 207us) server_negotiation.cc:407] Connection header received
0924 19:32:03.578187 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.578188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.578202 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.578260 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.579148 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.579884 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.581173 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.581358 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.581421 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.583727 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.583729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.583729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.583736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.585023 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.585421 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.585422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.585423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.585474 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.585641 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.585642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.585642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.585755 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.585817 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.586151 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.586232 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:32:03.586480 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.577433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42613 (local address 127.0.98.3:44991)
0924 19:32:03.577571 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.577606 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.578092 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.578373 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.578378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.578759 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:03.579049 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.579056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.580005 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.580006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.581090 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.581093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.581443 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.581445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.581565 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.582022 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:03.582030 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:03.583646 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:03.585123 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.585130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.585131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.585345 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.585515 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.585515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.585516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.585602 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.585834 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:03.585839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:03.585989 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.586119 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.586260 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260924 19:32:03.586898 8379 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42613
I20260924 19:32:03.586997 8379 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:03.587174 8379 heartbeater.cc:511] Master 127.0.98.62:42613 requested a full tablet report, sending...
I20260924 19:32:03.587541 7895 ts_manager.cc:194] Registered new tserver with Master: 6874673348a74969a155aa93342e167e (127.0.98.3:41567)
I20260924 19:32:03.587908 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44991
I20260924 19:32:03.592587 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.588198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45431 (local address 127.0.98.62:42613)
0924 19:32:03.588232 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.588233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.588303 (+ 70us) server_negotiation.cc:407] Connection header received
0924 19:32:03.588366 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.588366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.588373 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.588432 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.588810 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.589261 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.589892 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.590060 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.590102 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.590871 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.590872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.590872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.590877 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.591852 (+ 975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.592113 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.592114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.592114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.592156 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.592261 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.592261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.592262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.592342 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.592379 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.592477 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.592529 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 19:32:03.592603 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.588142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42613 (local address 127.0.98.3:45431)
0924 19:32:03.588253 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.588264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.588316 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.588441 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.588442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.588662 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:03.588730 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.588732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.589336 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.589336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.589841 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.589843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.590107 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.590108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.590151 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.590523 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:03.590528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:03.590808 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:03.591911 (+ 1103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.591911 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.591912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.592063 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.592190 (+ 127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.592190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.592191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.592230 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.592377 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:03.592377 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:03.592407 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.592463 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.592536 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260924 19:32:03.598029 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:32:03.620105 7965 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:03.615054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.0.98.62:42613)
0924 19:32:03.615092 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.615094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.615104 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:03.615163 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.615163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.615172 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.615210 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.615714 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.616215 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.616973 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.617185 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.617243 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.620021 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52842: BlockingRecv error: recv got EOF from 127.0.0.1:52842 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:32:03.626760 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.620799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.98.62:42613)
0924 19:32:03.620842 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.620844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.620898 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:32:03.620990 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.620991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.621000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.621039 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.621456 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.621913 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.622554 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.622728 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.622778 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.624755 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.624757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.624757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.624765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.625986 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.626256 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.626256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.626257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.626303 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.626423 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.626423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.626423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.626511 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.626577 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.626644 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.626694 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:32:03.628285 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:32:03.629885 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:32:03.636055 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.632061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.0.1:54566)
0924 19:32:03.632452 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.632470 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.632528 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.632836 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.632837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.632839 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:03.632892 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.632894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.633750 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.633751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.634919 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.634921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.635749 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.635750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.635837 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.635852 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.635925 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.635978 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":5,"thread_start_us":78,"threads_started":1}
I20260924 19:32:03.636096 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.632227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.0.98.1:36321)
0924 19:32:03.632370 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.632375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.632543 (+ 168us) server_negotiation.cc:407] Connection header received
0924 19:32:03.632609 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.632612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.632682 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.632753 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:03.632972 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.633672 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.634997 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.635670 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.635741 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.635817 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.635946 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.635990 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260924 19:32:03.636305 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.632084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32963 (local address 127.0.0.1:34744)
0924 19:32:03.632625 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.632653 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.632697 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.632895 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.632895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.632897 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:03.632927 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.632929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.634164 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.634165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.635147 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.635149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.636093 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.636094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.636157 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.636161 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.636206 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.636246 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":84,"threads_started":1}
I20260924 19:32:03.636487 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.632302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.0.98.2:32963)
0924 19:32:03.632432 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.632437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.632705 (+ 268us) server_negotiation.cc:407] Connection header received
0924 19:32:03.632752 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.632754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.632768 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.632833 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:03.632998 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.634079 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.635218 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.635929 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.635990 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.636061 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.636356 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.636389 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260924 19:32:03.637173 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.632060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41567 (local address 127.0.0.1:45720)
0924 19:32:03.632383 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.632448 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.632522 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.632797 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.632799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.632802 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:03.632883 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.632885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.635212 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.635213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.635917 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.635919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.636974 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.636975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.637038 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.637041 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.637085 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.637123 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":48,"threads_started":1}
I20260924 19:32:03.637108 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 7d93b10cc9da42e3b084a066c1819a1a (DEFAULT_TABLE table=test-table [id=9b93622b18014a1b8f8b19b66112bbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:03.637183 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 7d93b10cc9da42e3b084a066c1819a1a (DEFAULT_TABLE table=test-table [id=9b93622b18014a1b8f8b19b66112bbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:03.637458 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.632214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.0.98.3:41567)
0924 19:32:03.632455 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.632459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.632500 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:32:03.632611 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.632614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.632655 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.632720 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:03.632971 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.634083 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.636008 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.636797 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.636859 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.636931 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.637328 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.637366 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260924 19:32:03.637466 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d93b10cc9da42e3b084a066c1819a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.637466 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d93b10cc9da42e3b084a066c1819a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.637941 8312 tablet_service.cc:1467] Processing CreateTablet for tablet 7d93b10cc9da42e3b084a066c1819a1a (DEFAULT_TABLE table=test-table [id=9b93622b18014a1b8f8b19b66112bbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:03.638188 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d93b10cc9da42e3b084a066c1819a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.641065 8407 tablet_bootstrap.cc:492] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: Bootstrap starting.
I20260924 19:32:03.641207 8408 tablet_bootstrap.cc:492] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: Bootstrap starting.
I20260924 19:32:03.641848 8408 tablet_bootstrap.cc:654] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.641848 8407 tablet_bootstrap.cc:654] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.642168 8407 log.cc:824] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:03.642177 8408 log.cc:824] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:03.642516 8409 tablet_bootstrap.cc:492] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Bootstrap starting.
I20260924 19:32:03.642966 8407 tablet_bootstrap.cc:492] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: No bootstrap required, opened a new log
I20260924 19:32:03.643033 8407 ts_tablet_manager.cc:1397] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:32:03.643055 8408 tablet_bootstrap.cc:492] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: No bootstrap required, opened a new log
I20260924 19:32:03.643077 8409 tablet_bootstrap.cc:654] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.643100 8408 ts_tablet_manager.cc:1397] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:32:03.643368 8409 log.cc:824] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:03.644064 8409 tablet_bootstrap.cc:492] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: No bootstrap required, opened a new log
I20260924 19:32:03.644119 8409 ts_tablet_manager.cc:1397] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:32:03.644797 8408 raft_consensus.cc:348] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.644950 8408 raft_consensus.cc:374] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.644984 8408 raft_consensus.cc:729] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53ba118edac41aa99dfbef3d554d695, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.644941 8407 raft_consensus.cc:348] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.645114 8407 raft_consensus.cc:374] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.645148 8407 raft_consensus.cc:729] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6874673348a74969a155aa93342e167e, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.645117 8408 consensus_queue.cc:260] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.645256 8407 consensus_queue.cc:260] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.645347 8408 ts_tablet_manager.cc:1428] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:32:03.645454 8099 heartbeater.cc:503] Master 127.0.98.62:42613 was elected leader, sending a full tablet report...
I20260924 19:32:03.645500 8407 ts_tablet_manager.cc:1428] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:32:03.645627 8379 heartbeater.cc:503] Master 127.0.98.62:42613 was elected leader, sending a full tablet report...
I20260924 19:32:03.646075 8409 raft_consensus.cc:348] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.646227 8409 raft_consensus.cc:374] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.646268 8409 raft_consensus.cc:729] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dce50ac5140488a99bf57687e680b27, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.646368 8409 consensus_queue.cc:260] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.646610 8409 ts_tablet_manager.cc:1428] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:32:03.646741 8239 heartbeater.cc:503] Master 127.0.98.62:42613 was elected leader, sending a full tablet report...
W20260924 19:32:03.659135 8240 tablet.cc:2369] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:32:03.679057 8415 raft_consensus.cc:482] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:32:03.679169 8415 raft_consensus.cc:504] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.679453 8415 leader_election.cc:288] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6874673348a74969a155aa93342e167e (127.0.98.3:41567), f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321)
I20260924 19:32:03.682572 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.679767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51811 (local address 127.0.98.1:36321)
0924 19:32:03.679812 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.679814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.679823 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:32:03.679850 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.679851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.679862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.679911 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:03.680072 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.680887 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.681885 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.682241 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.682302 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.682313 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.682461 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.682501 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:32:03.682579 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.679763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46495 (local address 127.0.98.3:41567)
0924 19:32:03.679803 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.679805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.679934 (+ 129us) server_negotiation.cc:407] Connection header received
0924 19:32:03.679964 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.679965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.679976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.680022 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:03.680145 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.680905 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.681902 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.682256 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.682319 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.682329 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.682479 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.682519 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:32:03.682577 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.679651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.98.2:51811)
0924 19:32:03.679704 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.679717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.679770 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.679927 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.679928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.679930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:03.680006 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.680008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.680962 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.680963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.681797 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.681800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.682311 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.682312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.682398 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.682402 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.682440 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.682488 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260924 19:32:03.682602 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.679651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41567 (local address 127.0.98.2:46495)
0924 19:32:03.679863 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.679871 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.679922 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.680024 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.680025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.680026 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:03.680090 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.680091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.680990 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.680991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.681811 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.681814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.682331 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.682331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.682405 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.682408 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.682468 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.682506 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":15,"thread_start_us":115,"threads_started":1}
I20260924 19:32:03.683037 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93b10cc9da42e3b084a066c1819a1a" candidate_uuid: "0dce50ac5140488a99bf57687e680b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6874673348a74969a155aa93342e167e" is_pre_election: true
I20260924 19:32:03.683045 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93b10cc9da42e3b084a066c1819a1a" candidate_uuid: "0dce50ac5140488a99bf57687e680b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53ba118edac41aa99dfbef3d554d695" is_pre_election: true
I20260924 19:32:03.683198 8333 raft_consensus.cc:2454] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dce50ac5140488a99bf57687e680b27 in term 0.
I20260924 19:32:03.683198 8053 raft_consensus.cc:2454] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dce50ac5140488a99bf57687e680b27 in term 0.
I20260924 19:32:03.683686 8126 leader_election.cc:302] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dce50ac5140488a99bf57687e680b27, 6874673348a74969a155aa93342e167e; no voters:
I20260924 19:32:03.683807 8415 raft_consensus.cc:2781] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:32:03.683863 8415 raft_consensus.cc:482] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:32:03.683884 8415 raft_consensus.cc:3037] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.684587 8415 raft_consensus.cc:504] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.684743 8415 leader_election.cc:288] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [CANDIDATE]: Term 1 election: Requested vote from peers 6874673348a74969a155aa93342e167e (127.0.98.3:41567), f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321)
I20260924 19:32:03.684881 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93b10cc9da42e3b084a066c1819a1a" candidate_uuid: "0dce50ac5140488a99bf57687e680b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6874673348a74969a155aa93342e167e"
I20260924 19:32:03.684907 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d93b10cc9da42e3b084a066c1819a1a" candidate_uuid: "0dce50ac5140488a99bf57687e680b27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53ba118edac41aa99dfbef3d554d695"
I20260924 19:32:03.684968 8333 raft_consensus.cc:3037] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.684971 8053 raft_consensus.cc:3037] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.685804 8333 raft_consensus.cc:2454] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dce50ac5140488a99bf57687e680b27 in term 1.
I20260924 19:32:03.685814 8053 raft_consensus.cc:2454] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dce50ac5140488a99bf57687e680b27 in term 1.
I20260924 19:32:03.685957 8126 leader_election.cc:302] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dce50ac5140488a99bf57687e680b27, 6874673348a74969a155aa93342e167e; no voters:
I20260924 19:32:03.686040 8415 raft_consensus.cc:2781] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:03.686169 8415 raft_consensus.cc:686] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 1 LEADER]: Becoming Leader. State: Replica: 0dce50ac5140488a99bf57687e680b27, State: Running, Role: LEADER
I20260924 19:32:03.686244 8415 consensus_queue.cc:237] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.687018 7896 catalog_manager.cc:5495] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dce50ac5140488a99bf57687e680b27 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0dce50ac5140488a99bf57687e680b27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } health_report { overall_health: HEALTHY } } }
I20260924 19:32:03.696447 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.694503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.0.98.2:32963)
0924 19:32:03.694558 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.694559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.694565 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:32:03.694592 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.694593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.694608 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.694661 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:03.694761 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.695200 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.695896 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.696069 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.696109 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.696217 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:03.696306 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:03.696355 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.696368 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.696382 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:32:03.698014 8053 raft_consensus.cc:1261] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Refusing update from remote peer 0dce50ac5140488a99bf57687e680b27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:03.698017 8333 raft_consensus.cc:1261] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Refusing update from remote peer 0dce50ac5140488a99bf57687e680b27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:03.698314 8415 consensus_queue.cc:1035] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:03.698439 8417 consensus_queue.cc:1035] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:03.700687 8420 mvcc.cc:205] Tried to move back new op lower bound from 7332980013864779776 to 7332980013819437056. Current Snapshot: MvccSnapshot[applied={T|T < 7332980013864779776}]
I20260924 19:32:03.700985 8422 mvcc.cc:205] Tried to move back new op lower bound from 7332980013864779776 to 7332980013819437056. Current Snapshot: MvccSnapshot[applied={T|T < 7332980013864779776}]
I20260924 19:32:03.701138 8421 mvcc.cc:205] Tried to move back new op lower bound from 7332980013864779776 to 7332980013819437056. Current Snapshot: MvccSnapshot[applied={T|T < 7332980013864779776}]
I20260924 19:32:03.703506 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:03.703603 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:03.704258 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: Sending DeleteTablet for 3 replicas of tablet 7d93b10cc9da42e3b084a066c1819a1a
I20260924 19:32:03.704562 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d93b10cc9da42e3b084a066c1819a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45720
I20260924 19:32:03.704581 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d93b10cc9da42e3b084a066c1819a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34744
I20260924 19:32:03.704573 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d93b10cc9da42e3b084a066c1819a1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54566
I20260924 19:32:03.704794 8430 tablet_replica.cc:330] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: stopping tablet replica
I20260924 19:32:03.704890 8430 raft_consensus.cc:2229] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:03.704926 8431 tablet_replica.cc:330] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: stopping tablet replica
I20260924 19:32:03.705075 8431 raft_consensus.cc:2229] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:03.705020 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:32:03.705147 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:32:03.705297 8430 raft_consensus.cc:2258] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:03.705333 8431 raft_consensus.cc:2258] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:03.705879 8431 ts_tablet_manager.cc:1905] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:03.706004 8430 ts_tablet_manager.cc:1905] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:03.706487 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 1c952db8cffd48f2b5d47cc10634c8ca (DEFAULT_TABLE table=test-table [id=5ef35b8a8c95457697114ed105580986]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:03.706598 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c952db8cffd48f2b5d47cc10634c8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.706694 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 1c952db8cffd48f2b5d47cc10634c8ca (DEFAULT_TABLE table=test-table [id=5ef35b8a8c95457697114ed105580986]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:03.706912 8312 tablet_service.cc:1467] Processing CreateTablet for tablet 1c952db8cffd48f2b5d47cc10634c8ca (DEFAULT_TABLE table=test-table [id=5ef35b8a8c95457697114ed105580986]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:03.706996 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c952db8cffd48f2b5d47cc10634c8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.707775 8430 ts_tablet_manager.cc:1918] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:03.707882 8409 tablet_bootstrap.cc:492] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: Bootstrap starting.
I20260924 19:32:03.708269 8409 tablet_bootstrap.cc:654] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.708297 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c952db8cffd48f2b5d47cc10634c8ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.708149 8407 tablet_bootstrap.cc:492] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: Bootstrap starting.
I20260924 19:32:03.709108 8407 tablet_bootstrap.cc:654] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.709297 8408 tablet_bootstrap.cc:492] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: Bootstrap starting.
I20260924 19:32:03.709566 8409 tablet_bootstrap.cc:492] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: No bootstrap required, opened a new log
I20260924 19:32:03.709614 8409 ts_tablet_manager.cc:1397] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:32:03.709676 8408 tablet_bootstrap.cc:654] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.709753 8409 raft_consensus.cc:348] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.709812 8409 raft_consensus.cc:374] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.709833 8409 raft_consensus.cc:729] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dce50ac5140488a99bf57687e680b27, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.709872 8409 consensus_queue.cc:260] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.709962 8409 ts_tablet_manager.cc:1428] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.707835 8430 log.cc:1196] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/7d93b10cc9da42e3b084a066c1819a1a
I20260924 19:32:03.710585 8430 ts_tablet_manager.cc:1939] T 7d93b10cc9da42e3b084a066c1819a1a P 6874673348a74969a155aa93342e167e: Deleting consensus metadata
I20260924 19:32:03.710778 8431 ts_tablet_manager.cc:1918] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:03.710845 8431 log.cc:1196] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/7d93b10cc9da42e3b084a066c1819a1a
I20260924 19:32:03.710969 7882 catalog_manager.cc:4841] TS 6874673348a74969a155aa93342e167e (127.0.98.3:41567): tablet 7d93b10cc9da42e3b084a066c1819a1a (table test-table [id=9b93622b18014a1b8f8b19b66112bbe6]) successfully deleted
I20260924 19:32:03.711095 8431 ts_tablet_manager.cc:1939] T 7d93b10cc9da42e3b084a066c1819a1a P f53ba118edac41aa99dfbef3d554d695: Deleting consensus metadata
I20260924 19:32:03.711283 8407 tablet_bootstrap.cc:492] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: No bootstrap required, opened a new log
I20260924 19:32:03.711330 8407 ts_tablet_manager.cc:1397] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:32:03.711403 8408 tablet_bootstrap.cc:492] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: No bootstrap required, opened a new log
I20260924 19:32:03.711443 8408 ts_tablet_manager.cc:1397] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:32:03.711460 7881 catalog_manager.cc:4841] TS f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321): tablet 7d93b10cc9da42e3b084a066c1819a1a (table test-table [id=9b93622b18014a1b8f8b19b66112bbe6]) successfully deleted
I20260924 19:32:03.711453 8407 raft_consensus.cc:348] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.711515 8407 raft_consensus.cc:374] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.711534 8407 raft_consensus.cc:729] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6874673348a74969a155aa93342e167e, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.711570 8407 consensus_queue.cc:260] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.711619 8432 tablet_replica.cc:330] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: stopping tablet replica
I20260924 19:32:03.711591 8408 raft_consensus.cc:348] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.711650 8408 raft_consensus.cc:374] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.711649 8407 ts_tablet_manager.cc:1428] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.711671 8408 raft_consensus.cc:729] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53ba118edac41aa99dfbef3d554d695, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.711711 8432 raft_consensus.cc:2229] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:32:03.711705 8408 consensus_queue.cc:260] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.711778 8408 ts_tablet_manager.cc:1428] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.711848 8432 raft_consensus.cc:2258] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:03.712476 8432 ts_tablet_manager.cc:1905] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:03.713639 8432 ts_tablet_manager.cc:1918] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:03.713691 8432 log.cc:1196] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/7d93b10cc9da42e3b084a066c1819a1a
I20260924 19:32:03.713891 8432 ts_tablet_manager.cc:1939] T 7d93b10cc9da42e3b084a066c1819a1a P 0dce50ac5140488a99bf57687e680b27: Deleting consensus metadata
I20260924 19:32:03.714270 7883 catalog_manager.cc:4841] TS 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963): tablet 7d93b10cc9da42e3b084a066c1819a1a (table test-table [id=9b93622b18014a1b8f8b19b66112bbe6]) successfully deleted
W20260924 19:32:03.753614 8100 tablet.cc:2369] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:32:03.834365 8380 tablet.cc:2369] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:32:03.841486 8413 raft_consensus.cc:482] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:32:03.841578 8413 raft_consensus.cc:504] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.841928 8413 leader_election.cc:288] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963), 6874673348a74969a155aa93342e167e (127.0.98.3:41567)
I20260924 19:32:03.845153 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.842149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59731 (local address 127.0.98.3:41567)
0924 19:32:03.842192 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.842194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.842282 (+ 88us) server_negotiation.cc:407] Connection header received
0924 19:32:03.842330 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.842330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.842338 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.842435 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:03.842589 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.843340 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.844425 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.844824 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.844887 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.844898 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.845043 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.845086 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:32:03.845160 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.842146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59939 (local address 127.0.98.2:32963)
0924 19:32:03.842193 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.842195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.842282 (+ 87us) server_negotiation.cc:407] Connection header received
0924 19:32:03.842326 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.842327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.842338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.842424 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:03.842591 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.843341 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.844365 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.844822 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.844887 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.844897 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.845051 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.845095 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:32:03.845178 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.842077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32963 (local address 127.0.98.1:59939)
0924 19:32:03.842200 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.842210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.842274 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.842425 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.842426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.842429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:03.842527 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.842530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.843417 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.843417 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.844272 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.844275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.844902 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.844904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.844982 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.844986 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.845036 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.845084 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":10,"thread_start_us":62,"threads_started":1}
I20260924 19:32:03.845201 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.842077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41567 (local address 127.0.98.1:59731)
0924 19:32:03.842200 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.842210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.842274 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.842425 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.842426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.842431 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:03.842528 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.842530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.843416 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.843417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.844338 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.844342 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.844897 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.844898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.844973 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.844978 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.845022 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.845074 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":112,"thread_start_us":60,"threads_started":1}
I20260924 19:32:03.845492 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c952db8cffd48f2b5d47cc10634c8ca" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dce50ac5140488a99bf57687e680b27" is_pre_election: true
I20260924 19:32:03.845531 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c952db8cffd48f2b5d47cc10634c8ca" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6874673348a74969a155aa93342e167e" is_pre_election: true
I20260924 19:32:03.845618 8333 raft_consensus.cc:2454] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 0.
I20260924 19:32:03.845631 8193 raft_consensus.cc:2454] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 0.
I20260924 19:32:03.845772 7986 leader_election.cc:302] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6874673348a74969a155aa93342e167e, f53ba118edac41aa99dfbef3d554d695; no voters:
I20260924 19:32:03.845917 8413 raft_consensus.cc:2781] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:32:03.845960 8413 raft_consensus.cc:482] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:32:03.845974 8413 raft_consensus.cc:3037] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.846575 8413 raft_consensus.cc:504] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.846683 8413 leader_election.cc:288] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 election: Requested vote from peers 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963), 6874673348a74969a155aa93342e167e (127.0.98.3:41567)
I20260924 19:32:03.846834 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c952db8cffd48f2b5d47cc10634c8ca" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dce50ac5140488a99bf57687e680b27"
I20260924 19:32:03.846908 8193 raft_consensus.cc:3037] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.846875 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c952db8cffd48f2b5d47cc10634c8ca" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6874673348a74969a155aa93342e167e"
I20260924 19:32:03.846943 8333 raft_consensus.cc:3037] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.847486 8333 raft_consensus.cc:2454] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 1.
I20260924 19:32:03.847496 8193 raft_consensus.cc:2454] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 1.
I20260924 19:32:03.847637 7987 leader_election.cc:302] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dce50ac5140488a99bf57687e680b27, f53ba118edac41aa99dfbef3d554d695; no voters:
I20260924 19:32:03.847723 8413 raft_consensus.cc:2781] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:03.847844 8413 raft_consensus.cc:686] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 1 LEADER]: Becoming Leader. State: Replica: f53ba118edac41aa99dfbef3d554d695, State: Running, Role: LEADER
I20260924 19:32:03.847920 8413 consensus_queue.cc:237] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:03.848541 7896 catalog_manager.cc:5495] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 reported cstate change: term changed from 0 to 1, leader changed from <none> to f53ba118edac41aa99dfbef3d554d695 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } health_report { overall_health: UNKNOWN } } }
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:32:03.875937 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.869270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42613 (local address 127.0.0.1:52856)
0924 19:32:03.869415 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.869424 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.869491 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.869648 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.869649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.869949 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:03.870007 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.870009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.870609 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.870610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.871162 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.871164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.871494 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.871495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.871530 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.872016 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:03.872020 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:03.873336 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:03.874889 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.874897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.874899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.875125 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.875315 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:03.875316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:03.875316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:03.875382 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:03.875599 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:03.875606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:03.875655 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.875721 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.875830 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260924 19:32:03.875981 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.869321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.0.98.62:42613)
0924 19:32:03.869406 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.869408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.869482 (+ 74us) server_negotiation.cc:407] Connection header received
0924 19:32:03.869571 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.869572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.869581 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.869637 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.870074 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.870553 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.871241 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.871431 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.871495 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.873429 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:03.873431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:03.873431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:03.873437 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:03.874796 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.875205 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.875206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.875206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.875260 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:03.875428 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:03.875429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:03.875429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:03.875528 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:03.875574 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.875741 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.875872 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":58}
I20260924 19:32:03.877996 7896 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52856:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:32:03.878191 7896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:32:03.879539 8033 tablet_service.cc:1467] Processing CreateTablet for tablet f75ecb4a351c4ff48d6c6aa6c7aa7e76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1633da6166bc4174aee2c704cb6f9dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:32:03.879559 8173 tablet_service.cc:1467] Processing CreateTablet for tablet f75ecb4a351c4ff48d6c6aa6c7aa7e76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1633da6166bc4174aee2c704cb6f9dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:32:03.879685 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75ecb4a351c4ff48d6c6aa6c7aa7e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.879694 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75ecb4a351c4ff48d6c6aa6c7aa7e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.880839 8409 tablet_bootstrap.cc:492] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: Bootstrap starting.
I20260924 19:32:03.881012 8408 tablet_bootstrap.cc:492] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: Bootstrap starting.
I20260924 19:32:03.881116 8312 tablet_service.cc:1467] Processing CreateTablet for tablet f75ecb4a351c4ff48d6c6aa6c7aa7e76 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1633da6166bc4174aee2c704cb6f9dec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:32:03.881199 8409 tablet_bootstrap.cc:654] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.881254 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f75ecb4a351c4ff48d6c6aa6c7aa7e76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:03.881384 8408 tablet_bootstrap.cc:654] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.881735 8409 tablet_bootstrap.cc:492] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: No bootstrap required, opened a new log
I20260924 19:32:03.881783 8409 ts_tablet_manager.cc:1397] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:03.881899 8409 raft_consensus.cc:348] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.881976 8409 raft_consensus.cc:374] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.881999 8409 raft_consensus.cc:729] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dce50ac5140488a99bf57687e680b27, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.882016 8408 tablet_bootstrap.cc:492] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: No bootstrap required, opened a new log
I20260924 19:32:03.882057 8408 ts_tablet_manager.cc:1397] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:03.882040 8409 consensus_queue.cc:260] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.882128 8409 ts_tablet_manager.cc:1428] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:03.882186 8408 raft_consensus.cc:348] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.882243 8408 raft_consensus.cc:374] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.882215 8415 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.882262 8408 raft_consensus.cc:729] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53ba118edac41aa99dfbef3d554d695, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.882275 8415 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:32:03.882298 8408 consensus_queue.cc:260] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.882380 8408 ts_tablet_manager.cc:1428] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.882699 8407 tablet_bootstrap.cc:492] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: Bootstrap starting.
I20260924 19:32:03.882838 8413 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.882938 8413 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:32:03.883148 8407 tablet_bootstrap.cc:654] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:03.883903 8407 tablet_bootstrap.cc:492] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: No bootstrap required, opened a new log
I20260924 19:32:03.883948 8407 ts_tablet_manager.cc:1397] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:03.884076 8407 raft_consensus.cc:348] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.884127 8407 raft_consensus.cc:374] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:03.884140 8407 raft_consensus.cc:729] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6874673348a74969a155aa93342e167e, State: Initialized, Role: FOLLOWER
I20260924 19:32:03.884173 8407 consensus_queue.cc:260] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.884253 8407 ts_tablet_manager.cc:1428] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:03.884318 8414 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.884416 8414 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:32:03.922649 8413 raft_consensus.cc:482] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:32:03.922744 8413 raft_consensus.cc:504] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.922873 8413 leader_election.cc:288] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6874673348a74969a155aa93342e167e (127.0.98.3:41567), 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963)
I20260924 19:32:03.923094 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75ecb4a351c4ff48d6c6aa6c7aa7e76" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6874673348a74969a155aa93342e167e" is_pre_election: true
I20260924 19:32:03.923094 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75ecb4a351c4ff48d6c6aa6c7aa7e76" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dce50ac5140488a99bf57687e680b27" is_pre_election: true
I20260924 19:32:03.923202 8333 raft_consensus.cc:2454] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 0.
I20260924 19:32:03.923202 8193 raft_consensus.cc:2454] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 0.
I20260924 19:32:03.923343 7986 leader_election.cc:302] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6874673348a74969a155aa93342e167e, f53ba118edac41aa99dfbef3d554d695; no voters:
I20260924 19:32:03.923445 8413 raft_consensus.cc:2781] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:32:03.923496 8413 raft_consensus.cc:482] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:32:03.923521 8413 raft_consensus.cc:3037] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.924178 8413 raft_consensus.cc:504] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.924299 8413 leader_election.cc:288] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 election: Requested vote from peers 6874673348a74969a155aa93342e167e (127.0.98.3:41567), 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963)
I20260924 19:32:03.924445 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75ecb4a351c4ff48d6c6aa6c7aa7e76" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6874673348a74969a155aa93342e167e"
I20260924 19:32:03.924460 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f75ecb4a351c4ff48d6c6aa6c7aa7e76" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dce50ac5140488a99bf57687e680b27"
I20260924 19:32:03.924515 8333 raft_consensus.cc:3037] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.924521 8193 raft_consensus.cc:3037] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:03.925101 8333 raft_consensus.cc:2454] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 1.
I20260924 19:32:03.925170 8193 raft_consensus.cc:2454] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 1.
I20260924 19:32:03.925272 7986 leader_election.cc:302] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6874673348a74969a155aa93342e167e, f53ba118edac41aa99dfbef3d554d695; no voters:
I20260924 19:32:03.925347 8413 raft_consensus.cc:2781] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:03.925405 8413 raft_consensus.cc:686] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [term 1 LEADER]: Becoming Leader. State: Replica: f53ba118edac41aa99dfbef3d554d695, State: Running, Role: LEADER
I20260924 19:32:03.925449 8413 consensus_queue.cc:237] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } }
I20260924 19:32:03.925758 8438 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: TxnStatusTablet state changed. Reason: New leader f53ba118edac41aa99dfbef3d554d695. Latest consensus state: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.925882 8438 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:32:03.926064 8450 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:32:03.926105 7896 catalog_manager.cc:5495] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 reported cstate change: term changed from 0 to 1, leader changed from <none> to f53ba118edac41aa99dfbef3d554d695 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } health_report { overall_health: UNKNOWN } } }
I20260924 19:32:03.926199 8450 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:32:03.942400 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.940108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.0.98.1:36321)
0924 19:32:03.940164 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.940166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.940175 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:32:03.940238 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.940239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.940257 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.940310 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:03.940417 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.940967 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.941681 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.941891 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.941948 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.942110 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:03.942203 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:03.942261 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.942304 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.942324 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:32:03.942423 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.940045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.0.1:54570)
0924 19:32:03.940104 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.940116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.940159 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.940312 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.940313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.940314 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:32:03.940365 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.940367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.941045 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.941046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.941614 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.941616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.941952 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.941953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.942016 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.942032 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:32:03.942254 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:32:03.942261 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.942302 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.942350 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:32:03.943583 8333 raft_consensus.cc:1261] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Refusing update from remote peer f53ba118edac41aa99dfbef3d554d695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:03.943638 8193 raft_consensus.cc:1261] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Refusing update from remote peer f53ba118edac41aa99dfbef3d554d695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:03.943825 8438 consensus_queue.cc:1035] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:03.943938 8413 consensus_queue.cc:1035] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:03.944438 8415 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: TxnStatusTablet state changed. Reason: New leader f53ba118edac41aa99dfbef3d554d695. Latest consensus state: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.944561 8415 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:32:03.944881 8414 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: TxnStatusTablet state changed. Reason: New leader f53ba118edac41aa99dfbef3d554d695. Latest consensus state: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.944969 8414 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:32:03.945277 8413 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.945355 8413 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:32:03.945719 8413 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.945791 8413 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P f53ba118edac41aa99dfbef3d554d695: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:32:03.945935 8415 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.946018 8415 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 0dce50ac5140488a99bf57687e680b27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:32:03.946062 8414 tablet_replica.cc:439] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } }
I20260924 19:32:03.946131 8414 tablet_replica.cc:442] T f75ecb4a351c4ff48d6c6aa6c7aa7e76 P 6874673348a74969a155aa93342e167e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:32:03.949462 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.947386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.0.98.1:36321)
0924 19:32:03.947421 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.947422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.947465 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:32:03.947522 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.947522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.947532 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.947573 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:03.947688 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.948146 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.948882 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.949090 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.949141 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.949259 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:03.949290 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:03.949331 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.949385 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.949401 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:32:03.953812 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.951720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47521 (local address 127.0.98.1:36321)
0924 19:32:03.951773 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.951774 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.951782 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:32:03.951832 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.951833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.951840 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.951888 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:03.952010 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.952439 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.953211 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.953414 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.953463 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.953599 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:03.953626 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:03.953665 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:03.953711 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 19:32:03.953739 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":28}
I20260924 19:32:03.953884 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:03.951579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.98.1:47521)
0924 19:32:03.951728 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:03.951739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:03.951790 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:03.951891 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:03.951891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:03.951893 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:32:03.951964 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.951965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.952499 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.952500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.953143 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:03.953145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.953466 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.953467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:03.953525 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.953539 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:32:03.953670 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:32:03.953671 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:32:03.953711 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 19:32:03.953765 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":14,"thread_start_us":92,"threads_started":1}
I20260924 19:32:03.955919 8333 raft_consensus.cc:1261] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Refusing update from remote peer f53ba118edac41aa99dfbef3d554d695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:03.955951 8193 raft_consensus.cc:1261] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Refusing update from remote peer f53ba118edac41aa99dfbef3d554d695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:03.956130 8458 consensus_queue.cc:1035] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:03.956229 8413 consensus_queue.cc:1035] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:03.966812 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:03.966919 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:03.967445 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: Sending DeleteTablet for 3 replicas of tablet 1c952db8cffd48f2b5d47cc10634c8ca
I20260924 19:32:03.967667 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c952db8cffd48f2b5d47cc10634c8ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34744
I20260924 19:32:03.967760 8432 tablet_replica.cc:330] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: stopping tablet replica
I20260924 19:32:03.967729 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c952db8cffd48f2b5d47cc10634c8ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45720
I20260924 19:32:03.967777 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c952db8cffd48f2b5d47cc10634c8ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54566
I20260924 19:32:03.967808 8432 raft_consensus.cc:2229] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:03.967839 8432 raft_consensus.cc:2258] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:03.967859 8430 tablet_replica.cc:330] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: stopping tablet replica
I20260924 19:32:03.967905 8430 raft_consensus.cc:2229] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:03.967934 8430 raft_consensus.cc:2258] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:03.967991 8431 tablet_replica.cc:330] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: stopping tablet replica
I20260924 19:32:03.968046 8431 raft_consensus.cc:2229] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:32:03.968119 8431 raft_consensus.cc:2258] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:03.969959 8431 ts_tablet_manager.cc:1905] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:03.971689 8431 ts_tablet_manager.cc:1918] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:32:03.971742 8431 log.cc:1196] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/1c952db8cffd48f2b5d47cc10634c8ca
I20260924 19:32:03.971729 8430 ts_tablet_manager.cc:1905] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:03.971843 8432 ts_tablet_manager.cc:1905] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:03.971972 8431 ts_tablet_manager.cc:1939] T 1c952db8cffd48f2b5d47cc10634c8ca P f53ba118edac41aa99dfbef3d554d695: Deleting consensus metadata
I20260924 19:32:03.972395 7881 catalog_manager.cc:4841] TS f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321): tablet 1c952db8cffd48f2b5d47cc10634c8ca (table test-table [id=5ef35b8a8c95457697114ed105580986]) successfully deleted
I20260924 19:32:03.973346 8432 ts_tablet_manager.cc:1918] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:32:03.973387 8430 ts_tablet_manager.cc:1918] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:32:03.973407 8432 log.cc:1196] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/1c952db8cffd48f2b5d47cc10634c8ca
I20260924 19:32:03.973439 8430 log.cc:1196] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/1c952db8cffd48f2b5d47cc10634c8ca
I20260924 19:32:03.973618 8432 ts_tablet_manager.cc:1939] T 1c952db8cffd48f2b5d47cc10634c8ca P 0dce50ac5140488a99bf57687e680b27: Deleting consensus metadata
I20260924 19:32:03.973636 8430 ts_tablet_manager.cc:1939] T 1c952db8cffd48f2b5d47cc10634c8ca P 6874673348a74969a155aa93342e167e: Deleting consensus metadata
I20260924 19:32:03.973953 7883 catalog_manager.cc:4841] TS 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963): tablet 1c952db8cffd48f2b5d47cc10634c8ca (table test-table [id=5ef35b8a8c95457697114ed105580986]) successfully deleted
I20260924 19:32:03.974032 7882 catalog_manager.cc:4841] TS 6874673348a74969a155aa93342e167e (127.0.98.3:41567): tablet 1c952db8cffd48f2b5d47cc10634c8ca (table test-table [id=5ef35b8a8c95457697114ed105580986]) successfully deleted
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:32:03.999612 7965 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:03.993697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.0.98.62:42613)
0924 19:32:03.993761 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:03.993763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:03.993867 (+ 104us) server_negotiation.cc:407] Connection header received
0924 19:32:03.993954 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:03.993954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:03.993964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:03.994012 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:03.994688 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.995191 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.996226 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:03.996416 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:03.996470 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:03.999547 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52858: BlockingRecv error: recv got EOF from 127.0.0.1:52858 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 19:32:03 dist-test-slave-m8jb krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 19:32:04.007196 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.000399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.0.98.62:42613)
0924 19:32:04.000454 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:04.000455 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:04.000492 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:32:04.000553 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:04.000554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:04.000563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:04.000609 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:04.001050 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.001521 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.002172 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.002366 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.002425 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.004732 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:04.004734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:04.004734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:04.004741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:04.006050 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:04.006359 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:04.006360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:04.006361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:04.006432 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:04.006599 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:04.006600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:04.006600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:04.006742 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:04.006846 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:04.007008 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 19:32:04.007099 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:32:04.008879 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:32:04.009064 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:32:04.010577 8032 tablet_service.cc:1467] Processing CreateTablet for tablet c3269498d95243c5853265cd2807420a (DEFAULT_TABLE table=test-table [id=9fc442c259ce421583c9a79a39e21b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:04.010571 8173 tablet_service.cc:1467] Processing CreateTablet for tablet c3269498d95243c5853265cd2807420a (DEFAULT_TABLE table=test-table [id=9fc442c259ce421583c9a79a39e21b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:04.010597 8312 tablet_service.cc:1467] Processing CreateTablet for tablet c3269498d95243c5853265cd2807420a (DEFAULT_TABLE table=test-table [id=9fc442c259ce421583c9a79a39e21b05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:04.010723 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3269498d95243c5853265cd2807420a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:04.010723 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3269498d95243c5853265cd2807420a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:04.010723 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3269498d95243c5853265cd2807420a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:04.012138 8409 tablet_bootstrap.cc:492] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: Bootstrap starting.
I20260924 19:32:04.012140 8407 tablet_bootstrap.cc:492] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: Bootstrap starting.
I20260924 19:32:04.012233 8408 tablet_bootstrap.cc:492] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: Bootstrap starting.
I20260924 19:32:04.012521 8409 tablet_bootstrap.cc:654] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:04.012715 8408 tablet_bootstrap.cc:654] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:04.013075 8407 tablet_bootstrap.cc:654] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:04.013553 8408 tablet_bootstrap.cc:492] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: No bootstrap required, opened a new log
I20260924 19:32:04.013602 8408 ts_tablet_manager.cc:1397] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:32:04.013666 8407 tablet_bootstrap.cc:492] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: No bootstrap required, opened a new log
I20260924 19:32:04.013711 8407 ts_tablet_manager.cc:1397] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:32:04.013765 8409 tablet_bootstrap.cc:492] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: No bootstrap required, opened a new log
I20260924 19:32:04.013743 8408 raft_consensus.cc:348] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.013813 8409 ts_tablet_manager.cc:1397] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:32:04.013823 8408 raft_consensus.cc:374] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:04.013844 8408 raft_consensus.cc:729] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53ba118edac41aa99dfbef3d554d695, State: Initialized, Role: FOLLOWER
I20260924 19:32:04.013885 8408 consensus_queue.cc:260] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.013862 8407 raft_consensus.cc:348] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.013932 8407 raft_consensus.cc:374] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:04.013952 8407 raft_consensus.cc:729] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6874673348a74969a155aa93342e167e, State: Initialized, Role: FOLLOWER
I20260924 19:32:04.013911 8409 raft_consensus.cc:348] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.013969 8408 ts_tablet_manager.cc:1428] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:04.013975 8409 raft_consensus.cc:374] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:04.013996 8409 raft_consensus.cc:729] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dce50ac5140488a99bf57687e680b27, State: Initialized, Role: FOLLOWER
I20260924 19:32:04.013999 8407 consensus_queue.cc:260] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.014038 8409 consensus_queue.cc:260] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.014106 8407 ts_tablet_manager.cc:1428] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 19:32:04.014124 8409 ts_tablet_manager.cc:1428] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:04.026314 8413 raft_consensus.cc:482] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:32:04.026403 8413 raft_consensus.cc:504] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.026549 8413 leader_election.cc:288] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963), 6874673348a74969a155aa93342e167e (127.0.98.3:41567)
I20260924 19:32:04.026749 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3269498d95243c5853265cd2807420a" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dce50ac5140488a99bf57687e680b27" is_pre_election: true
I20260924 19:32:04.026767 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3269498d95243c5853265cd2807420a" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6874673348a74969a155aa93342e167e" is_pre_election: true
I20260924 19:32:04.026849 8333 raft_consensus.cc:2454] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 0.
I20260924 19:32:04.026849 8193 raft_consensus.cc:2454] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 0.
I20260924 19:32:04.026997 7987 leader_election.cc:302] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dce50ac5140488a99bf57687e680b27, f53ba118edac41aa99dfbef3d554d695; no voters:
I20260924 19:32:04.027076 8413 raft_consensus.cc:2781] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:32:04.027105 8413 raft_consensus.cc:482] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:32:04.027123 8413 raft_consensus.cc:3037] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:04.027733 8413 raft_consensus.cc:504] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.027842 8413 leader_election.cc:288] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 election: Requested vote from peers 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963), 6874673348a74969a155aa93342e167e (127.0.98.3:41567)
I20260924 19:32:04.027995 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3269498d95243c5853265cd2807420a" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dce50ac5140488a99bf57687e680b27"
I20260924 19:32:04.028019 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3269498d95243c5853265cd2807420a" candidate_uuid: "f53ba118edac41aa99dfbef3d554d695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6874673348a74969a155aa93342e167e"
I20260924 19:32:04.028081 8193 raft_consensus.cc:3037] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:04.028112 8333 raft_consensus.cc:3037] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:04.028755 8193 raft_consensus.cc:2454] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 1.
I20260924 19:32:04.028915 7987 leader_election.cc:302] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dce50ac5140488a99bf57687e680b27, f53ba118edac41aa99dfbef3d554d695; no voters:
I20260924 19:32:04.028951 8333 raft_consensus.cc:2454] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53ba118edac41aa99dfbef3d554d695 in term 1.
I20260924 19:32:04.029003 8413 raft_consensus.cc:2781] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:04.029047 8413 raft_consensus.cc:686] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 1 LEADER]: Becoming Leader. State: Replica: f53ba118edac41aa99dfbef3d554d695, State: Running, Role: LEADER
I20260924 19:32:04.029095 8413 consensus_queue.cc:237] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.029672 7895 catalog_manager.cc:5495] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 reported cstate change: term changed from 0 to 1, leader changed from <none> to f53ba118edac41aa99dfbef3d554d695 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f53ba118edac41aa99dfbef3d554d695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } health_report { overall_health: UNKNOWN } } }
I20260924 19:32:04.039002 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.036449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.0.98.1:36321)
0924 19:32:04.036502 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:04.036504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:04.036514 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:04.036561 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:04.036562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:04.036573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:04.036622 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:04.036785 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.037513 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.038285 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.038603 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.038666 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.038787 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:04.038820 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:04.038866 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:04.038902 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 19:32:04.038921 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:32:04.039819 8193 raft_consensus.cc:1261] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Refusing update from remote peer f53ba118edac41aa99dfbef3d554d695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:04.039845 8333 raft_consensus.cc:1261] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Refusing update from remote peer f53ba118edac41aa99dfbef3d554d695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:04.040053 8458 consensus_queue.cc:1035] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:04.040150 8413 consensus_queue.cc:1035] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:04.042665 7896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:04.042762 7896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:04.043351 7896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: Sending DeleteTablet for 3 replicas of tablet c3269498d95243c5853265cd2807420a
I20260924 19:32:04.043563 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet c3269498d95243c5853265cd2807420a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34744
I20260924 19:32:04.043654 8432 tablet_replica.cc:330] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: stopping tablet replica
I20260924 19:32:04.043699 8432 raft_consensus.cc:2229] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:04.043733 8432 raft_consensus.cc:2258] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:04.043916 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet c3269498d95243c5853265cd2807420a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54566
I20260924 19:32:04.044003 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet c3269498d95243c5853265cd2807420a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54566
I20260924 19:32:04.044059 8431 tablet_replica.cc:330] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: stopping tablet replica
I20260924 19:32:04.044047 8432 ts_tablet_manager.cc:1905] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:04.044109 8431 raft_consensus.cc:2229] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:32:04.044162 8431 raft_consensus.cc:2258] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:04.044323 7896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:32:04.044456 8431 ts_tablet_manager.cc:1905] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:32:04.044451 7896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:32:04.044732 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet c3269498d95243c5853265cd2807420a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45720
I20260924 19:32:04.044804 8430 tablet_replica.cc:330] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: stopping tablet replica
I20260924 19:32:04.044838 8430 raft_consensus.cc:2229] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:04.044873 8430 raft_consensus.cc:2258] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:04.045825 8173 tablet_service.cc:1467] Processing CreateTablet for tablet ac28c4ab796a490e8d8be748a59868d7 (DEFAULT_TABLE table=test-table [id=18a85a105da843b4a0788de8b2239c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:04.045936 8430 ts_tablet_manager.cc:1905] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:04.045938 8431 ts_tablet_manager.cc:1918] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:04.045981 8431 log.cc:1196] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/c3269498d95243c5853265cd2807420a
I20260924 19:32:04.046152 8431 ts_tablet_manager.cc:1939] T c3269498d95243c5853265cd2807420a P f53ba118edac41aa99dfbef3d554d695: Deleting consensus metadata
I20260924 19:32:04.046311 8432 ts_tablet_manager.cc:1918] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:04.046353 8432 log.cc:1196] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/c3269498d95243c5853265cd2807420a
I20260924 19:32:04.046357 8033 tablet_service.cc:1467] Processing CreateTablet for tablet ac28c4ab796a490e8d8be748a59868d7 (DEFAULT_TABLE table=test-table [id=18a85a105da843b4a0788de8b2239c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:04.046452 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28c4ab796a490e8d8be748a59868d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:04.046571 8432 ts_tablet_manager.cc:1939] T c3269498d95243c5853265cd2807420a P 0dce50ac5140488a99bf57687e680b27: Deleting consensus metadata
I20260924 19:32:04.047199 8430 ts_tablet_manager.cc:1918] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:04.047247 8430 log.cc:1196] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/c3269498d95243c5853265cd2807420a
I20260924 19:32:04.047396 8430 ts_tablet_manager.cc:1939] T c3269498d95243c5853265cd2807420a P 6874673348a74969a155aa93342e167e: Deleting consensus metadata
I20260924 19:32:04.047536 8408 tablet_bootstrap.cc:492] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695: Bootstrap starting.
I20260924 19:32:04.047572 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28c4ab796a490e8d8be748a59868d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:04.047894 8408 tablet_bootstrap.cc:654] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:04.048372 8408 tablet_bootstrap.cc:492] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695: No bootstrap required, opened a new log
I20260924 19:32:04.048411 8408 ts_tablet_manager.cc:1397] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:32:04.048532 8408 raft_consensus.cc:348] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.048583 8408 raft_consensus.cc:374] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:04.048599 8408 raft_consensus.cc:729] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53ba118edac41aa99dfbef3d554d695, State: Initialized, Role: FOLLOWER
I20260924 19:32:04.048658 8408 consensus_queue.cc:260] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.048688 8409 tablet_bootstrap.cc:492] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: Bootstrap starting.
I20260924 19:32:04.048744 8408 ts_tablet_manager.cc:1428] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:04.048966 8409 tablet_bootstrap.cc:654] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:04.049268 7883 catalog_manager.cc:4841] TS 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963): tablet c3269498d95243c5853265cd2807420a (table test-table [id=9fc442c259ce421583c9a79a39e21b05]) successfully deleted
I20260924 19:32:04.049773 8409 tablet_bootstrap.cc:492] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: No bootstrap required, opened a new log
I20260924 19:32:04.049827 8409 ts_tablet_manager.cc:1397] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:32:04.049927 8409 raft_consensus.cc:348] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.049996 8409 raft_consensus.cc:374] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:04.050024 8409 raft_consensus.cc:729] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dce50ac5140488a99bf57687e680b27, State: Initialized, Role: FOLLOWER
I20260924 19:32:04.050068 8409 consensus_queue.cc:260] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.050159 8409 ts_tablet_manager.cc:1428] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 19:32:04.050663 7881 catalog_manager.cc:4841] TS f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321): tablet c3269498d95243c5853265cd2807420a (table test-table [id=9fc442c259ce421583c9a79a39e21b05]) successfully deleted
W20260924 19:32:04.050783 7881 catalog_manager.cc:4808] TS f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321): delete failed for tablet c3269498d95243c5853265cd2807420a because the tablet was not found. No further retry: Not found: Tablet not found: c3269498d95243c5853265cd2807420a
I20260924 19:32:04.050932 7882 catalog_manager.cc:4841] TS 6874673348a74969a155aa93342e167e (127.0.98.3:41567): tablet c3269498d95243c5853265cd2807420a (table test-table [id=9fc442c259ce421583c9a79a39e21b05]) successfully deleted
I20260924 19:32:04.051009 8312 tablet_service.cc:1467] Processing CreateTablet for tablet ac28c4ab796a490e8d8be748a59868d7 (DEFAULT_TABLE table=test-table [id=18a85a105da843b4a0788de8b2239c84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:04.051103 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28c4ab796a490e8d8be748a59868d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:04.052326 8407 tablet_bootstrap.cc:492] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: Bootstrap starting.
I20260924 19:32:04.052734 8407 tablet_bootstrap.cc:654] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:04.053439 8407 tablet_bootstrap.cc:492] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: No bootstrap required, opened a new log
I20260924 19:32:04.053479 8407 ts_tablet_manager.cc:1397] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 19:32:04.053592 8407 raft_consensus.cc:348] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.053651 8407 raft_consensus.cc:374] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:04.053663 8407 raft_consensus.cc:729] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6874673348a74969a155aa93342e167e, State: Initialized, Role: FOLLOWER
I20260924 19:32:04.053694 8407 consensus_queue.cc:260] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.053776 8407 ts_tablet_manager.cc:1428] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:04.175390 8414 raft_consensus.cc:482] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:32:04.175490 8414 raft_consensus.cc:504] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.175773 8414 leader_election.cc:288] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321), 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963)
I20260924 19:32:04.179665 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.176034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37007 (local address 127.0.98.2:32963)
0924 19:32:04.176088 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:04.176090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:04.176200 (+ 110us) server_negotiation.cc:407] Connection header received
0924 19:32:04.176237 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:04.176239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:04.176252 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:04.176312 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:04.176448 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.177228 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.178761 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.179302 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.179372 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.179382 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:04.179556 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 19:32:04.179592 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:32:04.179665 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.176014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52983 (local address 127.0.98.1:36321)
0924 19:32:04.176089 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:04.176091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:04.176168 (+ 77us) server_negotiation.cc:407] Connection header received
0924 19:32:04.176241 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:04.176242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:04.176249 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:04.176325 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:04.176456 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.177195 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.178753 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.179302 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.179372 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.179382 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:04.179556 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 19:32:04.179590 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:32:04.179718 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.175966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.98.3:52983)
0924 19:32:04.176099 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:04.176109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:04.176183 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:04.176322 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:04.176322 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:04.176325 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:04.176399 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:04.176400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.177310 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.177311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:04.178654 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:04.178658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.179398 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.179399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:04.179480 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.179487 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:32:04.179547 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:32:04.179601 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":105,"thread_start_us":63,"threads_started":1}
I20260924 19:32:04.179721 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.175970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32963 (local address 127.0.98.3:37007)
0924 19:32:04.176133 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:04.176142 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:04.176191 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:04.176323 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:04.176323 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:04.176325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:04.176381 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:04.176384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.177310 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.177311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:04.178654 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:04.178658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.179397 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.179399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:04.179480 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.179485 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:32:04.179547 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:32:04.179601 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":14,"thread_start_us":63,"threads_started":1}
I20260924 19:32:04.180115 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac28c4ab796a490e8d8be748a59868d7" candidate_uuid: "6874673348a74969a155aa93342e167e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dce50ac5140488a99bf57687e680b27" is_pre_election: true
I20260924 19:32:04.180208 8193 raft_consensus.cc:2454] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6874673348a74969a155aa93342e167e in term 0.
I20260924 19:32:04.180374 8267 leader_election.cc:302] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dce50ac5140488a99bf57687e680b27, 6874673348a74969a155aa93342e167e; no voters:
I20260924 19:32:04.180526 8414 raft_consensus.cc:2781] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:32:04.180579 8414 raft_consensus.cc:482] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:32:04.180598 8414 raft_consensus.cc:3037] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:04.180603 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac28c4ab796a490e8d8be748a59868d7" candidate_uuid: "6874673348a74969a155aa93342e167e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53ba118edac41aa99dfbef3d554d695" is_pre_election: true
I20260924 19:32:04.180702 8053 raft_consensus.cc:2454] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6874673348a74969a155aa93342e167e in term 0.
I20260924 19:32:04.181317 8414 raft_consensus.cc:504] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.181433 8414 leader_election.cc:288] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [CANDIDATE]: Term 1 election: Requested vote from peers f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321), 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963)
I20260924 19:32:04.181559 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac28c4ab796a490e8d8be748a59868d7" candidate_uuid: "6874673348a74969a155aa93342e167e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dce50ac5140488a99bf57687e680b27"
I20260924 19:32:04.181579 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac28c4ab796a490e8d8be748a59868d7" candidate_uuid: "6874673348a74969a155aa93342e167e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53ba118edac41aa99dfbef3d554d695"
I20260924 19:32:04.181638 8053 raft_consensus.cc:3037] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:04.181634 8193 raft_consensus.cc:3037] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:04.182165 8053 raft_consensus.cc:2454] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6874673348a74969a155aa93342e167e in term 1.
I20260924 19:32:04.182235 8193 raft_consensus.cc:2454] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6874673348a74969a155aa93342e167e in term 1.
I20260924 19:32:04.182317 8265 leader_election.cc:302] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6874673348a74969a155aa93342e167e, f53ba118edac41aa99dfbef3d554d695; no voters:
I20260924 19:32:04.182397 8414 raft_consensus.cc:2781] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:04.182500 8414 raft_consensus.cc:686] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 1 LEADER]: Becoming Leader. State: Replica: 6874673348a74969a155aa93342e167e, State: Running, Role: LEADER
I20260924 19:32:04.182566 8414 consensus_queue.cc:237] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } }
I20260924 19:32:04.183215 7896 catalog_manager.cc:5495] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6874673348a74969a155aa93342e167e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6874673348a74969a155aa93342e167e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6874673348a74969a155aa93342e167e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41567 } health_report { overall_health: HEALTHY } } }
I20260924 19:32:04.203545 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.201405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.0.98.3:41567)
0924 19:32:04.201444 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:04.201446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:04.201456 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:04.201525 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:04.201525 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:04.201544 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:04.201616 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:04.201696 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.202181 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.202865 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.203048 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.203108 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.203278 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:04.203380 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:04.203435 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:04.203470 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 19:32:04.203485 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:32:04.208158 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.205927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41175 (local address 127.0.98.1:36321)
0924 19:32:04.205981 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:04.205983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:04.205992 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:32:04.206025 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:04.206026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:04.206035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:04.206076 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:04.206190 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.206792 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.207531 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.207730 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.207783 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.207937 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:04.207968 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:04.208008 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:04.208065 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 19:32:04.208083 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:32:04.208178 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.205788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.98.3:41175)
0924 19:32:04.205913 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:04.205923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:04.205970 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:04.206082 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:04.206082 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:04.206084 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:32:04.206138 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:04.206139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.206875 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.206876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:04.207463 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:04.207464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.207795 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.207797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:04.207860 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.207873 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:32:04.208017 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:32:04.208019 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:32:04.208053 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 19:32:04.208095 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":50,"threads_started":1}
I20260924 19:32:04.210052 8053 raft_consensus.cc:1261] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Refusing update from remote peer 6874673348a74969a155aa93342e167e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:04.210184 8193 raft_consensus.cc:1261] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Refusing update from remote peer 6874673348a74969a155aa93342e167e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:04.210271 8487 consensus_queue.cc:1035] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f53ba118edac41aa99dfbef3d554d695" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:04.210404 8414 consensus_queue.cc:1035] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dce50ac5140488a99bf57687e680b27" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:04.216691 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.214696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44857 (local address 127.0.98.3:41567)
0924 19:32:04.214728 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:04.214730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:04.214737 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:32:04.214779 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:04.214780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:04.214788 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:04.214831 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:04.214915 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.215376 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.216043 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.216255 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.216314 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.216462 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:04.216493 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:04.216536 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:04.216579 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:32:04.216598 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:32:04.216717 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.214644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41567 (local address 127.0.98.1:44857)
0924 19:32:04.214681 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:04.214692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:04.214728 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:04.214832 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:04.214833 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:04.214834 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:32:04.214874 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:04.214875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.215444 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.215445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:04.215975 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:04.215977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:04.216322 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:04.216323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:04.216394 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:04.216400 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:32:04.216536 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:32:04.216537 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 19:32:04.216577 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 19:32:04.216624 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 19:32:04.222970 7896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:04.223065 7896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:04.223662 7896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59ff2a98d540461a95b8be58e6574c62: Sending DeleteTablet for 3 replicas of tablet ac28c4ab796a490e8d8be748a59868d7
I20260924 19:32:04.223904 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet ac28c4ab796a490e8d8be748a59868d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45720
I20260924 19:32:04.224015 8430 tablet_replica.cc:330] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: stopping tablet replica
I20260924 19:32:04.224001 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet ac28c4ab796a490e8d8be748a59868d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34744
I20260924 19:32:04.224085 8430 raft_consensus.cc:2229] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:32:04.224093 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet ac28c4ab796a490e8d8be748a59868d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54566
I20260924 19:32:04.224152 8430 raft_consensus.cc:2258] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:04.224166 8432 tablet_replica.cc:330] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: stopping tablet replica
I20260924 19:32:04.224216 8432 raft_consensus.cc:2229] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:04.224247 8432 raft_consensus.cc:2258] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:04.224299 8431 tablet_replica.cc:330] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695: stopping tablet replica
I20260924 19:32:04.224350 8431 raft_consensus.cc:2229] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:04.224387 8431 raft_consensus.cc:2258] T ac28c4ab796a490e8d8be748a59868d7 P f53ba118edac41aa99dfbef3d554d695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:04.224467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 7968
I20260924 19:32:04.225337 8432 ts_tablet_manager.cc:1905] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:04.226876 8432 ts_tablet_manager.cc:1918] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:32:04.226934 8432 log.cc:1196] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/ac28c4ab796a490e8d8be748a59868d7
I20260924 19:32:04.226979 8430 ts_tablet_manager.cc:1905] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:04.227139 8432 ts_tablet_manager.cc:1939] T ac28c4ab796a490e8d8be748a59868d7 P 0dce50ac5140488a99bf57687e680b27: Deleting consensus metadata
I20260924 19:32:04.227697 7883 catalog_manager.cc:4841] TS 0dce50ac5140488a99bf57687e680b27 (127.0.98.2:32963): tablet ac28c4ab796a490e8d8be748a59868d7 (table test-table [id=18a85a105da843b4a0788de8b2239c84]) successfully deleted
I20260924 19:32:04.228276 8430 ts_tablet_manager.cc:1918] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:32:04.228327 8430 log.cc:1196] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/ac28c4ab796a490e8d8be748a59868d7
I20260924 19:32:04.228502 8430 ts_tablet_manager.cc:1939] T ac28c4ab796a490e8d8be748a59868d7 P 6874673348a74969a155aa93342e167e: Deleting consensus metadata
I20260924 19:32:04.228948 7882 catalog_manager.cc:4841] TS 6874673348a74969a155aa93342e167e (127.0.98.3:41567): tablet ac28c4ab796a490e8d8be748a59868d7 (table test-table [id=18a85a105da843b4a0788de8b2239c84]) successfully deleted
W20260924 19:32:04.232116 7881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36321 (error 108)
I20260924 19:32:04.232375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 8108
I20260924 19:32:04.237736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 8248
I20260924 19:32:04.237787 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:04.232458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36321 (local address 127.0.0.1:54612)
0924 19:32:04.237706 (+ 5248us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:04.237748 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5230,"thread_start_us":39,"threads_started":1}
W20260924 19:32:04.239382 7881 catalog_manager.cc:4584] TS f53ba118edac41aa99dfbef3d554d695 (127.0.98.1:36321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac28c4ab796a490e8d8be748a59868d7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36321: connect: Connection refused (error 111)
I20260924 19:32:04.242564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 7864
2026-09-24T19:32:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2387 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:32:04 dist-test-slave-m8jb krb5kdc[8500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:32:04 dist-test-slave-m8jb krb5kdc[8500](info): set up 2 sockets
Sep 24 19:32:04 dist-test-slave-m8jb krb5kdc[8500](info): commencing operation
krb5kdc: starting...
W20260924 19:32:05.188879 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:32:05 dist-test-slave-m8jb krb5kdc[8500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:32:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:32:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:32:05.303946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:05.383567 8513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:05.383781 8513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:05.383805 8513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:05.385015 8513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:32:05.385059 8513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:05.385072 8513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:05.385083 8513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:32:05.385093 8513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:32:05.386595 8513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:05.386797 8513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:05.387005 8513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:05.391808 8521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:05.391907 8513 server_base.cc:1034] running on GCE node
W20260924 19:32:05.391825 8519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:05.391794 8518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:05.392256 8513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:05.392549 8513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:05.393697 8513 hybrid_clock.cc:648] HybridClock initialized: now 1790278325393680 us; error 33 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T19:32:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1159 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:32:05 dist-test-slave-m8jb krb5kdc[8526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:32:05 dist-test-slave-m8jb krb5kdc[8526](info): set up 2 sockets
Sep 24 19:32:05 dist-test-slave-m8jb krb5kdc[8526](info): commencing operation
krb5kdc: starting...
W20260924 19:32:06.348801 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:32:06 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:32:06.460250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39053
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:06.538398 8539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:06.538640 8539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:06.538669 8539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:06.539817 8539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:32:06.539856 8539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:06.539873 8539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:06.539888 8539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:32:06.539904 8539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:32:06.541420 8539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:06.541659 8539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:06.541853 8539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:06.546329 8545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:06.546361 8547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:06.546535 8539 server_base.cc:1034] running on GCE node
W20260924 19:32:06.546393 8544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:06.546815 8539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:06.547096 8539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:06.548226 8539 hybrid_clock.cc:648] HybridClock initialized: now 1790278326548216 us; error 25 us; skew 500 ppm
Sep 24 19:32:06 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:06.550570 8539 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:32:06.550957 8539 webserver.cc:460] Webserver started at http://127.0.98.62:37323/ using document root <none> and password file <none>
I20260924 19:32:06.551180 8539 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:06.551229 8539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:06.551347 8539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:06.552368 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "d5c53c8dd4d54a9997f64b5bb11540a2"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "b269e590473eb10a6202a25f66bced0d"
server_key_iv: "266ccfdb6fe08c256558ce69feb1d8c5"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.552736 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "d5c53c8dd4d54a9997f64b5bb11540a2"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "b269e590473eb10a6202a25f66bced0d"
server_key_iv: "266ccfdb6fe08c256558ce69feb1d8c5"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.553972 8539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:06.554678 8554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:06.554823 8539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:06.554913 8539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "d5c53c8dd4d54a9997f64b5bb11540a2"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "b269e590473eb10a6202a25f66bced0d"
server_key_iv: "266ccfdb6fe08c256558ce69feb1d8c5"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.554975 8539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:06.573863 8539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:06.575953 8539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:06.576093 8539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:06.579737 8539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39053
I20260924 19:32:06.579777 8606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39053 every 8 connection(s)
I20260924 19:32:06.580245 8539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:32:06.580973 8607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:06.583137 8607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Bootstrap starting.
I20260924 19:32:06.583693 8607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:06.583930 8607 log.cc:824] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:06.584616 8607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: No bootstrap required, opened a new log
I20260924 19:32:06.584887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 8539
I20260924 19:32:06.584981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:32:06.585137 392 external_mini_cluster.cc:1428] Setting key 9843cfba6d149b20482888754c96c727
I20260924 19:32:06.587033 8607 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260924 19:32:06.587194 8607 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:06.587230 8607 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c53c8dd4d54a9997f64b5bb11540a2, State: Initialized, Role: FOLLOWER
I20260924 19:32:06.587363 8607 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260924 19:32:06.587440 8607 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:32:06.587478 8607 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:32:06.587523 8607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:06.588147 8607 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260924 19:32:06.588228 8607 leader_election.cc:302] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c53c8dd4d54a9997f64b5bb11540a2; no voters:
I20260924 19:32:06.588392 8607 leader_election.cc:288] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:32:06.588435 8612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:06.588573 8612 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 1 LEADER]: Becoming Leader. State: Replica: d5c53c8dd4d54a9997f64b5bb11540a2, State: Running, Role: LEADER
I20260924 19:32:06.588665 8607 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:32:06.588728 8612 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260924 19:32:06.589074 8612 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c53c8dd4d54a9997f64b5bb11540a2. Latest consensus state: current_term: 1 leader_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } } }
I20260924 19:32:06.589123 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } } }
I20260924 19:32:06.589236 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:06.589447 8612 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:06.589618 8621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:32:06.590171 8621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:32:06.591678 8621 catalog_manager.cc:1347] Generated new cluster ID: 453e9609702d465dbf1dd180132d5787
I20260924 19:32:06.591718 8621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:32:06 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:32:06.596947 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:06.585605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.0.98.62:39053)
0924 19:32:06.585753 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:06.585760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:06.585784 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:32:06.586959 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:06.586966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:06.587027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:06.587142 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:06.587908 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.589087 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.590121 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.590684 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.590821 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:06.593457 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:06.593462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:06.593462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:06.593476 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:06.595360 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:06.595724 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:06.595727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:06.595727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:06.595848 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:06.596026 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:06.596026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:06.596026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:06.596325 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:06.596470 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:06.596581 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:32:06.596747 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 19:32:06.606267 8621 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:32:06.606827 8621 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:32:06.615409 8621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Generated new TSK 0
I20260924 19:32:06.615613 8621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:32:06.645548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:06.724815 8633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:06.725008 8633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:06.725029 8633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:06.726109 8633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:06.726140 8633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:06.726162 8633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:32:06.727573 8633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:06.727742 8633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:06.727916 8633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:06.732523 8639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:06.732427 8641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:06.732609 8633 server_base.cc:1034] running on GCE node
W20260924 19:32:06.732441 8638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:06.732995 8633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:06.733222 8633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:06.734360 8633 hybrid_clock.cc:648] HybridClock initialized: now 1790278326734347 us; error 25 us; skew 500 ppm
Sep 24 19:32:06 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:06.740088 8633 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:32:06.740496 8633 webserver.cc:460] Webserver started at http://127.0.98.1:34717/ using document root <none> and password file <none>
I20260924 19:32:06.740717 8633 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:06.740763 8633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:06.740885 8633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:06.741889 8633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "b88201421c5c4dbdb3dba97a177f295b"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "78b651805cea42efa9b1eda2d0047971"
server_key_iv: "f53b578b2b0f7eb23879b5f48edfd449"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.742229 8633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b88201421c5c4dbdb3dba97a177f295b"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "78b651805cea42efa9b1eda2d0047971"
server_key_iv: "f53b578b2b0f7eb23879b5f48edfd449"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.743479 8633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:32:06.744155 8648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:06.744300 8633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:32:06.744390 8633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "b88201421c5c4dbdb3dba97a177f295b"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "78b651805cea42efa9b1eda2d0047971"
server_key_iv: "f53b578b2b0f7eb23879b5f48edfd449"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.744450 8633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:06.757798 8633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:06.759771 8633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:06.759903 8633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:06.760111 8633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:06.760401 8633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:06.760433 8633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:06.760462 8633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:06.760488 8633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:06.765810 8633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34993
I20260924 19:32:06.765857 8761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34993 every 8 connection(s)
I20260924 19:32:06.766160 8633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:32:06.769837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 8633
I20260924 19:32:06.769922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:32:06.770094 392 external_mini_cluster.cc:1428] Setting key 529c7baa76c068c5839bc788fa2e535b
Sep 24 19:32:06 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:32:06.775167 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:06.766805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39043 (local address 127.0.98.62:39053)
0924 19:32:06.766883 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:06.766886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:06.767040 (+ 154us) server_negotiation.cc:407] Connection header received
0924 19:32:06.767901 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:06.767902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:06.767912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:06.767961 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:06.768543 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.769099 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.770084 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.770350 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.770418 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:06.772566 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:06.772568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:06.772568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:06.772576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:06.773930 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:06.774293 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:06.774294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:06.774295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:06.774357 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:06.774564 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:06.774565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:06.774565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:06.774704 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:06.774772 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:06.774985 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 19:32:06.775071 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 19:32:06.775233 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:06.766775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:39043)
0924 19:32:06.766916 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:06.766958 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:06.767820 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:06.768020 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:06.768024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:06.768268 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:06.768465 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:06.768470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.769172 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.769172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:06.770006 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:06.770008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.770495 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.770496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:06.770613 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:06.771098 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:06.771107 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:06.772476 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:06.774010 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:06.774016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:06.774018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:06.774227 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:06.774416 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:06.774416 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:06.774417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:06.774513 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:06.774759 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:06.774762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:06.774873 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 19:32:06.774947 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:32:06.775070 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260924 19:32:06.775686 8762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39053
I20260924 19:32:06.775770 8762 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:06.775940 8762 heartbeater.cc:511] Master 127.0.98.62:39053 requested a full tablet report, sending...
I20260924 19:32:06.776497 8571 ts_manager.cc:194] Registered new tserver with Master: b88201421c5c4dbdb3dba97a177f295b (127.0.98.1:34993)
I20260924 19:32:06.777426 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39043
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:32:06.795173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:06.873104 8767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:06.873302 8767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:06.873332 8767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:06.874461 8767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:06.874501 8767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:06.874536 8767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:32:06.875989 8767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:06.876207 8767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:06.876405 8767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:06.881033 8772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:06.881032 8775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:06.881140 8767 server_base.cc:1034] running on GCE node
W20260924 19:32:06.881093 8773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:06.881480 8767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:06.881709 8767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:06.882851 8767 hybrid_clock.cc:648] HybridClock initialized: now 1790278326882834 us; error 26 us; skew 500 ppm
Sep 24 19:32:06 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:06.885288 8767 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:32:06.885674 8767 webserver.cc:460] Webserver started at http://127.0.98.2:35363/ using document root <none> and password file <none>
I20260924 19:32:06.885840 8767 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:06.885879 8767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:06.885968 8767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:06.886945 8767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd63d7f1aa5e4773acf252d496cc21c9"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "e91b51d17bf54fdc58c021ffcef7933e"
server_key_iv: "0379794821765602c8903c3e7521c197"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.887244 8767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd63d7f1aa5e4773acf252d496cc21c9"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "e91b51d17bf54fdc58c021ffcef7933e"
server_key_iv: "0379794821765602c8903c3e7521c197"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.888412 8767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:32:06.889192 8782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:06.889353 8767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:32:06.889438 8767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "fd63d7f1aa5e4773acf252d496cc21c9"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "e91b51d17bf54fdc58c021ffcef7933e"
server_key_iv: "0379794821765602c8903c3e7521c197"
server_key_version: "encryptionkey@0"
I20260924 19:32:06.889511 8767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:06.899019 8767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:06.901376 8767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:06.901525 8767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:06.901739 8767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:06.902045 8767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:06.902081 8767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:06.902114 8767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:06.902143 8767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:06.908011 8767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35527
I20260924 19:32:06.908066 8895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35527 every 8 connection(s)
I20260924 19:32:06.908385 8767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:32:06.909502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 8767
I20260924 19:32:06.909597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:32:06.909793 392 external_mini_cluster.cc:1428] Setting key c3317bfb51df65f672ea0bd5e4ddb914
Sep 24 19:32:06 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:32:06.917975 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:06.909096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49829 (local address 127.0.98.62:39053)
0924 19:32:06.909145 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:06.909147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:06.909317 (+ 170us) server_negotiation.cc:407] Connection header received
0924 19:32:06.910358 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:06.910360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:06.910371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:06.910439 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:06.911034 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.911562 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.912914 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.913197 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.913267 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:06.915292 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:06.915293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:06.915293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:06.915301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:06.916733 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:06.917118 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:06.917120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:06.917121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:06.917178 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:06.917373 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:06.917374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:06.917374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:06.917505 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:06.917570 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:06.917826 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 19:32:06.917906 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:32:06.918081 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:06.909052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:49829)
0924 19:32:06.909210 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:06.909250 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:06.910267 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:06.910509 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:06.910513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:06.910758 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:06.910948 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:06.910952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.911663 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.911664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:06.912832 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:06.912835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:06.913273 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:06.913274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:06.913371 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:06.913888 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:06.913897 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:06.915202 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:06.916828 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:06.916834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:06.916836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:06.917047 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:06.917236 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:06.917236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:06.917236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:06.917321 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:06.917557 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:06.917560 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:06.917684 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 19:32:06.917775 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:32:06.917899 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 19:32:06.918493 8896 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39053
I20260924 19:32:06.918603 8896 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:06.918769 8896 heartbeater.cc:511] Master 127.0.98.62:39053 requested a full tablet report, sending...
I20260924 19:32:06.919143 8571 ts_manager.cc:194] Registered new tserver with Master: fd63d7f1aa5e4773acf252d496cc21c9 (127.0.98.2:35527)
I20260924 19:32:06.919600 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:32:06.935982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:07.015235 8901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:07.015460 8901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:07.015489 8901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:07.016700 8901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:07.016741 8901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:07.016776 8901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:32:07.018296 8901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:07.018548 8901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:07.018774 8901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:07.023679 8907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.023679 8906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.023697 8909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:07.024017 8901 server_base.cc:1034] running on GCE node
I20260924 19:32:07.024240 8901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:07.024477 8901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:07.025622 8901 hybrid_clock.cc:648] HybridClock initialized: now 1790278327025599 us; error 39 us; skew 500 ppm
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:07.028046 8901 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:32:07.028450 8901 webserver.cc:460] Webserver started at http://127.0.98.3:34961/ using document root <none> and password file <none>
I20260924 19:32:07.028678 8901 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:07.028731 8901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:07.028855 8901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:07.029858 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ae35cbd5d1845be8241c5639597e283"
format_stamp: "Formatted at 2026-09-24 19:32:07 on dist-test-slave-m8jb"
server_key: "eccc41b5e8dfba8ab3fa3e7e1c9b1063"
server_key_iv: "c49cb96cfd901b98146c393aa89161ef"
server_key_version: "encryptionkey@0"
I20260924 19:32:07.030212 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ae35cbd5d1845be8241c5639597e283"
format_stamp: "Formatted at 2026-09-24 19:32:07 on dist-test-slave-m8jb"
server_key: "eccc41b5e8dfba8ab3fa3e7e1c9b1063"
server_key_iv: "c49cb96cfd901b98146c393aa89161ef"
server_key_version: "encryptionkey@0"
I20260924 19:32:07.031436 8901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:07.032169 8916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.032354 8901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:07.032434 8901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "3ae35cbd5d1845be8241c5639597e283"
format_stamp: "Formatted at 2026-09-24 19:32:07 on dist-test-slave-m8jb"
server_key: "eccc41b5e8dfba8ab3fa3e7e1c9b1063"
server_key_iv: "c49cb96cfd901b98146c393aa89161ef"
server_key_version: "encryptionkey@0"
I20260924 19:32:07.032501 8901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:07.041062 8901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:07.043073 8901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:07.043203 8901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:07.043403 8901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:07.043689 8901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:07.043721 8901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.043749 8901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:07.043772 8901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.049075 8901 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35465
I20260924 19:32:07.049124 9029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35465 every 8 connection(s)
I20260924 19:32:07.049420 8901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:32:07.050467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 8901
I20260924 19:32:07.050550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:32:07.050715 392 external_mini_cluster.cc:1428] Setting key c6e66b9fc2f590a099d0145436b13a49
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:32:07.058434 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.050155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34913 (local address 127.0.98.62:39053)
0924 19:32:07.050219 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.050222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.050391 (+ 169us) server_negotiation.cc:407] Connection header received
0924 19:32:07.051464 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.051465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.051476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.051517 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.052132 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.052623 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.053648 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.053841 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.053894 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.055916 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:07.055917 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:07.055918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:07.055924 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:07.057203 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:07.057560 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:07.057561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:07.057562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:07.057614 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:07.057819 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:07.057820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:07.057820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:07.057925 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:07.057982 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:07.058275 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 19:32:07.058368 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:32:07.058558 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.050104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:34913)
0924 19:32:07.050276 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.050316 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.051372 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.051597 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.051602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.051848 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.052039 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.052043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.052746 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.052748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.053571 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.053574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.053904 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.053905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.054036 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.054459 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.054467 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.055828 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:07.057274 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:07.057281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:07.057283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:07.057483 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:07.057667 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:07.057667 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:07.057668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:07.057767 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:07.057978 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:07.057984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:07.058117 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 19:32:07.058225 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:32:07.058362 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260924 19:32:07.058993 9030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39053
I20260924 19:32:07.059082 9030 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:07.059242 9030 heartbeater.cc:511] Master 127.0.98.62:39053 requested a full tablet report, sending...
I20260924 19:32:07.059561 8571 ts_manager.cc:194] Registered new tserver with Master: 3ae35cbd5d1845be8241c5639597e283 (127.0.98.3:35465)
I20260924 19:32:07.059886 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34913
I20260924 19:32:07.062371 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 19:32:07.131605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 8633
I20260924 19:32:07.135607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 8767
I20260924 19:32:07.139012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 8901
I20260924 19:32:07.142635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 8539
I20260924 19:32:07.146724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39053
--webserver_interface=127.0.98.62
--webserver_port=37323
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:07.225184 9039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:07.225396 9039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:07.225417 9039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:07.226586 9039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:32:07.226629 9039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:07.226642 9039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:07.226655 9039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:32:07.226665 9039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:32:07.228156 9039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:07.228387 9039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:07.228595 9039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:07.233170 9045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.233212 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.233194 9044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:07.233507 9039 server_base.cc:1034] running on GCE node
I20260924 19:32:07.233723 9039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:07.233971 9039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:07.235116 9039 hybrid_clock.cc:648] HybridClock initialized: now 1790278327235102 us; error 28 us; skew 500 ppm
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:07.237509 9039 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:32:07.237985 9039 webserver.cc:460] Webserver started at http://127.0.98.62:37323/ using document root <none> and password file <none>
I20260924 19:32:07.238186 9039 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:07.238245 9039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:07.239676 9039 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:32:07.240377 9054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.240592 9039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:32:07.240701 9039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "d5c53c8dd4d54a9997f64b5bb11540a2"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "b269e590473eb10a6202a25f66bced0d"
server_key_iv: "266ccfdb6fe08c256558ce69feb1d8c5"
server_key_version: "encryptionkey@0"
I20260924 19:32:07.241077 9039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:07.259011 9039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:07.260804 9039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:07.260942 9039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:07.264436 9039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39053
I20260924 19:32:07.264485 9106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39053 every 8 connection(s)
I20260924 19:32:07.264981 9039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:32:07.266207 9107 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:32:07.266829 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Bootstrap starting.
I20260924 19:32:07.271102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 9039
I20260924 19:32:07.271438 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34993
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39053
--webserver_port=34717
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:32:07.274981 9107 log.cc:824] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:07.276603 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:32:07.276887 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Bootstrap complete.
I20260924 19:32:07.278956 9107 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260924 19:32:07.279183 9107 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c53c8dd4d54a9997f64b5bb11540a2, State: Initialized, Role: FOLLOWER
I20260924 19:32:07.279323 9107 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260924 19:32:07.279388 9107 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:32:07.279433 9107 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:32:07.279476 9107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:32:07.280424 9107 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260924 19:32:07.280511 9107 leader_election.cc:302] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c53c8dd4d54a9997f64b5bb11540a2; no voters:
I20260924 19:32:07.280680 9107 leader_election.cc:288] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:32:07.280731 9111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:32:07.280849 9111 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [term 2 LEADER]: Becoming Leader. State: Replica: d5c53c8dd4d54a9997f64b5bb11540a2, State: Running, Role: LEADER
I20260924 19:32:07.280937 9107 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:32:07.280928 9111 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } }
I20260924 19:32:07.281234 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c53c8dd4d54a9997f64b5bb11540a2. Latest consensus state: current_term: 2 leader_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } } }
I20260924 19:32:07.281314 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:07.281507 9113 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c53c8dd4d54a9997f64b5bb11540a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39053 } } }
I20260924 19:32:07.281600 9113 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:07.281725 9116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:32:07.282199 9116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:32:07.282313 9116 catalog_manager.cc:1259] Loaded cluster ID: 453e9609702d465dbf1dd180132d5787
I20260924 19:32:07.282342 9116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:32:07.283645 9116 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:32:07.283814 9116 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d5c53c8dd4d54a9997f64b5bb11540a2: Loaded TSK: 0
I20260924 19:32:07.284027 9116 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:32:07.351723 9109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:07.351926 9109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:07.351948 9109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:07.353179 9109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:07.353220 9109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:07.353246 9109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:32:07.354708 9109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34717
--tserver_master_addrs=127.0.98.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:07.354920 9109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:07.355131 9109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:07.359822 9133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:07.359978 9109 server_base.cc:1034] running on GCE node
W20260924 19:32:07.359865 9136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.360075 9134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:07.360275 9109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:07.360488 9109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:07.361631 9109 hybrid_clock.cc:648] HybridClock initialized: now 1790278327361617 us; error 31 us; skew 500 ppm
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:07.364094 9109 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 19:32:07.364485 9109 webserver.cc:460] Webserver started at http://127.0.98.1:34717/ using document root <none> and password file <none>
I20260924 19:32:07.364717 9109 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:07.364781 9109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:07.366226 9109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:07.366782 9143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.366931 9109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:07.367019 9109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "b88201421c5c4dbdb3dba97a177f295b"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "78b651805cea42efa9b1eda2d0047971"
server_key_iv: "f53b578b2b0f7eb23879b5f48edfd449"
server_key_version: "encryptionkey@0"
I20260924 19:32:07.367352 9109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:07.382738 9109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:07.384847 9109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:07.384964 9109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:07.385195 9109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:07.385510 9109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:07.385543 9109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.385565 9109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:07.385576 9109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.391371 9109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34993
I20260924 19:32:07.391427 9256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34993 every 8 connection(s)
I20260924 19:32:07.391747 9109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:32:07.396297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 9109
I20260924 19:32:07.396580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35527
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39053
--webserver_port=35363
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:07.401582 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.392397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:50381)
0924 19:32:07.392563 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.392612 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.393239 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.393850 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.393854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.394387 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.394585 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.394589 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.396140 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.396141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.397329 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.397332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.398170 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.398171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.398312 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.398882 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.398892 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.401379 (+ 2487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
W20260924 19:32:07.401728 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.392485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50381 (local address 127.0.98.62:39053)
0924 19:32:07.393425 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.393434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.393469 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:32:07.393602 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.393609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.393667 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.393803 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.394690 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.396051 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.397414 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.398057 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.398169 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.401543 (+ 3374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50381: BlockingRecv error: recv got EOF from 127.0.98.1:50381 (error 108)
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":75,"threads_started":1}
W20260924 19:32:07.402065 9257 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39053 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39053: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:07.407377 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.402270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:33999)
0924 19:32:07.402369 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.402378 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.402404 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.402541 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.402541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.402683 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.402734 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.402735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.403599 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.403600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.404356 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.404359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.404857 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.404858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.404923 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.405380 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.405385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.407308 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84}
W20260924 19:32:07.407650 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.402331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33999 (local address 127.0.98.62:39053)
0924 19:32:07.402384 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.402386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.402415 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:32:07.402459 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.402460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.402474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.402537 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.402798 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.403522 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.404440 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.404785 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.404858 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.407392 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33999: BlockingRecv error: recv got EOF from 127.0.98.1:33999 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:07.411347 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.407617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:53919)
0924 19:32:07.407652 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.407662 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.407700 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.407862 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.407862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.407969 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.407994 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.407995 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.408559 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.408560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.409100 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.409101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.409391 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.409392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.409437 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.409808 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.409811 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.411283 (+ 1472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 19:32:07.411397 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.407673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53919 (local address 127.0.98.62:39053)
0924 19:32:07.407749 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.407751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.407757 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:32:07.407776 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.407776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.407786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.407829 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.408066 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.408496 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.409173 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.409337 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.409388 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.411348 (+ 1960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53919: BlockingRecv error: recv got EOF from 127.0.98.1:53919 (error 108)
Metrics: {"server-negotiator.queue_time_us":63,"thread_start_us":92,"threads_started":1}
W20260924 19:32:07.411456 9257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:32:07.477129 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:07.477326 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:07.477348 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:07.478524 9261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:07.478590 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:07.478616 9261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:32:07.480070 9261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35363
--tserver_master_addrs=127.0.98.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:07.480296 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:07.480509 9261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:07.485399 9267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.485404 9270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.485387 9268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:07.485606 9261 server_base.cc:1034] running on GCE node
I20260924 19:32:07.485795 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:07.486037 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:07.487172 9261 hybrid_clock.cc:648] HybridClock initialized: now 1790278327487155 us; error 28 us; skew 500 ppm
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:07.489468 9261 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 19:32:07.489877 9261 webserver.cc:460] Webserver started at http://127.0.98.2:35363/ using document root <none> and password file <none>
I20260924 19:32:07.490088 9261 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:07.490149 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:07.491470 9261 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:32:07.491918 9277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.492070 9261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.492147 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "fd63d7f1aa5e4773acf252d496cc21c9"
format_stamp: "Formatted at 2026-09-24 19:32:06 on dist-test-slave-m8jb"
server_key: "e91b51d17bf54fdc58c021ffcef7933e"
server_key_iv: "0379794821765602c8903c3e7521c197"
server_key_version: "encryptionkey@0"
I20260924 19:32:07.492456 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:07.518353 9261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:07.520503 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:07.520669 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:07.520897 9261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:07.521198 9261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:07.521232 9261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.521260 9261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:07.521284 9261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.526775 9261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35527
I20260924 19:32:07.526829 9390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35527 every 8 connection(s)
I20260924 19:32:07.527159 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:32:07.531654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 9261
I20260924 19:32:07.531960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35465
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39053
--webserver_port=34961
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:32:07.534366 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.527813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57035 (local address 127.0.98.62:39053)
0924 19:32:07.527869 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.527871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.528066 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:32:07.528519 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.528521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.528534 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.528581 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.529495 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.529974 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.530905 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.531085 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.531129 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.534317 (+ 3188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57035: BlockingRecv error: recv got EOF from 127.0.98.2:57035 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:32:07.534432 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.527802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:57035)
0924 19:32:07.527946 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.527982 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.528445 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.528667 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.528671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.529230 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.529424 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.529429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.530040 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.530040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.530847 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.530849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.531138 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.531139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.531240 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.531745 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.531756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.534230 (+ 2474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
W20260924 19:32:07.534626 9391 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39053 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39053: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:07.538568 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.534822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:52725)
0924 19:32:07.534866 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.534874 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.534903 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.535045 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.535045 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.535157 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.535211 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.535213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.535810 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.535811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.536308 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.536309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.536615 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.536617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.536701 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.537105 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.537109 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.538497 (+ 1388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 19:32:07.538647 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.534885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52725 (local address 127.0.98.62:39053)
0924 19:32:07.534938 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.534940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.534947 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:32:07.534968 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.534969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.534978 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.535021 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.535283 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.535748 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.536374 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.536560 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.536611 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.538598 (+ 1987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52725: BlockingRecv error: recv got EOF from 127.0.98.2:52725 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:07.542783 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.539228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:39333)
0924 19:32:07.539266 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.539274 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.539300 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.539430 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.539431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.539536 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.539561 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.539562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.540119 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.540120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.540586 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.540588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.540895 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.540896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.540943 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.541310 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.541314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.542720 (+ 1406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:32:07.542834 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.539304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39333 (local address 127.0.98.62:39053)
0924 19:32:07.539351 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.539353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.539359 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:32:07.539374 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.539375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.539382 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.539422 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.539619 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.540058 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.540670 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.540843 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.540893 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.542788 (+ 1895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39333: BlockingRecv error: recv got EOF from 127.0.98.2:39333 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 19:32:07.542896 9391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:32:07.614122 9394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:07.614390 9394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:07.614423 9394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:07.615899 9394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:07.615994 9394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:07.616048 9394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:32:07.617635 9394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34961
--tserver_master_addrs=127.0.98.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:07.617857 9394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:07.618078 9394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:07.622805 9399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.622792 9400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:07.622862 9402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:07.622895 9394 server_base.cc:1034] running on GCE node
I20260924 19:32:07.623242 9394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:07.623468 9394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:07.624614 9394 hybrid_clock.cc:648] HybridClock initialized: now 1790278327624598 us; error 29 us; skew 500 ppm
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278327, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:07.627005 9394 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 19:32:07.627419 9394 webserver.cc:460] Webserver started at http://127.0.98.3:34961/ using document root <none> and password file <none>
I20260924 19:32:07.627627 9394 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:07.627694 9394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:07.629015 9394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:32:07.629526 9409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.629663 9394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.629739 9394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "3ae35cbd5d1845be8241c5639597e283"
format_stamp: "Formatted at 2026-09-24 19:32:07 on dist-test-slave-m8jb"
server_key: "eccc41b5e8dfba8ab3fa3e7e1c9b1063"
server_key_iv: "c49cb96cfd901b98146c393aa89161ef"
server_key_version: "encryptionkey@0"
I20260924 19:32:07.630096 9394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:07.652602 9394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:07.654798 9394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:07.654943 9394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:07.655174 9394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:07.655476 9394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:07.655509 9394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.655539 9394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:07.655555 9394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:07.660835 9394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35465
I20260924 19:32:07.660894 9522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35465 every 8 connection(s)
I20260924 19:32:07.661212 9394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:07.666888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 9394
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:32:07.668174 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.661864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51831 (local address 127.0.98.62:39053)
0924 19:32:07.661910 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.661912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.662132 (+ 220us) server_negotiation.cc:407] Connection header received
0924 19:32:07.662640 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.662641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.662654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.662705 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.663611 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.664094 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.665095 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.665279 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.665334 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.668118 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51831: BlockingRecv error: recv got EOF from 127.0.98.3:51831 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:32:07.668239 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.661846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:51831)
0924 19:32:07.662012 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.662048 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.662540 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.662776 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.662780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.663322 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.663516 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.663520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.664171 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.664173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.665012 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.665015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.665345 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.665346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.665465 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.665921 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.665930 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.668047 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
W20260924 19:32:07.668622 9523 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39053 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39053: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:07.672695 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.668869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:58641)
0924 19:32:07.668907 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.668915 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.668944 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.669048 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.669049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.669187 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.669234 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.669235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.669843 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.669844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.670504 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.670507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.670799 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.670800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.670863 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.671189 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.671193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.672613 (+ 1420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:32:07.672781 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.668905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58641 (local address 127.0.98.62:39053)
0924 19:32:07.668942 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.668943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.668951 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:32:07.668993 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.668993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.669000 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.669044 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.669292 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.669772 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.670570 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.670736 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.670782 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.672726 (+ 1944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58641: BlockingRecv error: recv got EOF from 127.0.98.3:58641 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:32:07.674033 9262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.667397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.98.62:39053)
0924 19:32:07.667450 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.667452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.667522 (+ 70us) server_negotiation.cc:407] Connection header received
0924 19:32:07.667580 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.667581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.667592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.667656 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.668170 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.668893 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.669747 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.669968 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.670030 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.671273 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:07.671277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:07.671277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:07.671287 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:07.672770 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:07.673145 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:07.673147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:07.673148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:07.673236 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:07.673384 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:07.673384 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:07.673385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:07.673622 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:07.673749 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:07.673854 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:32:07.673955 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:07 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:07.676587 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:07.672891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:47357)
0924 19:32:07.672998 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:07.673006 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:07.673048 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:07.673293 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:07.673293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:07.673391 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:07.673413 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.673414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.673966 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.673967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.674603 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:07.674605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.674943 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.674944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:07.674984 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.675234 (+ 250us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:07.675238 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:07.676544 (+ 1306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
W20260924 19:32:07.676714 9523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:32:07.676738 9260 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:07.672952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47357 (local address 127.0.98.62:39053)
0924 19:32:07.672993 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:07.672995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:07.673054 (+ 59us) server_negotiation.cc:407] Connection header received
0924 19:32:07.673095 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:07.673095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:07.673104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:07.673146 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:07.673477 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.673905 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.674672 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:07.674873 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:07.674924 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:07.676687 (+ 1763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47357: BlockingRecv error: recv got EOF from 127.0.98.3:47357 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 19:32:08 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:08 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:08.416109 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:08.411839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:43753)
0924 19:32:08.412000 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:08.412014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:08.412079 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:08.412240 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:08.412241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:08.412398 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:08.412447 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:08.412449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.413100 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.413101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:08.413636 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:08.413638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.413940 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.413941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:08.414000 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:08.414445 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:08.414449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:08.416020 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
W20260924 19:32:08.416201 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:08.411958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43753 (local address 127.0.98.62:39053)
0924 19:32:08.412068 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:08.412070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:08.412079 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:32:08.412134 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:08.412135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:08.412144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:08.412192 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:08.412509 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.413039 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.413702 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.413892 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.413937 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:08.416139 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43753: BlockingRecv error: recv got EOF from 127.0.98.1:43753 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
Sep 24 19:32:08 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:08 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:08.547690 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:08.543287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:52335)
0924 19:32:08.543426 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:08.543437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:08.543509 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:08.543662 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:08.543662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:08.543850 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:08.543899 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:08.543901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.544594 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.544595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:08.545188 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:08.545191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.545529 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.545530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:08.545586 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:08.546021 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:08.546026 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:08.547610 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
W20260924 19:32:08.547785 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:08.543355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52335 (local address 127.0.98.62:39053)
0924 19:32:08.543420 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:08.543423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:08.543497 (+ 74us) server_negotiation.cc:407] Connection header received
0924 19:32:08.543573 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:08.543574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:08.543587 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:08.543650 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:08.543971 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.544505 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.545272 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.545462 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.545518 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:08.547723 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52335: BlockingRecv error: recv got EOF from 127.0.98.2:52335 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 19:32:08 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:08 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:08.682030 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:08.677156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:55871)
0924 19:32:08.677309 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:08.677321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:08.677400 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:08.677531 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:08.677532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:08.677717 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:08.677763 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:08.677764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.678572 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.678573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:08.679113 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:08.679114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.679447 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.679448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:08.679512 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:08.679945 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:08.679950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:08.681954 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
W20260924 19:32:08.682078 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:08.677268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55871 (local address 127.0.98.62:39053)
0924 19:32:08.677326 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:08.677328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:08.677394 (+ 66us) server_negotiation.cc:407] Connection header received
0924 19:32:08.677454 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:08.677455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:08.677468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:08.677518 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:08.677836 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.678494 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.679192 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:08.679389 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:08.679443 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:08.682028 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55871: BlockingRecv error: recv got EOF from 127.0.98.3:55871 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 19:32:09 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:09 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:09.420811 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:09.416547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:42423)
0924 19:32:09.416695 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:09.416706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:09.416766 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:09.416880 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:09.416880 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:09.417053 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:09.417103 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:09.417105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.417717 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.417718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:09.418254 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:09.418255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.418638 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.418639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:09.418679 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:09.419089 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:09.419094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:09.420746 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
W20260924 19:32:09.420919 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:09.416605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42423 (local address 127.0.98.62:39053)
0924 19:32:09.416732 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:09.416733 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:09.416768 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:32:09.416807 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:09.416807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:09.416817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:09.416869 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:09.417157 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.417653 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.418345 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.418569 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.418631 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:09.420849 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42423: BlockingRecv error: recv got EOF from 127.0.98.1:42423 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
Sep 24 19:32:09 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:09 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:09.552399 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:09.548113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:41485)
0924 19:32:09.548244 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:09.548256 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:09.548320 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:09.548455 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:09.548456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:09.548656 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:09.548713 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:09.548716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.549349 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.549350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:09.549878 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:09.549880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.550195 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.550196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:09.550249 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:09.550664 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:09.550669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:09.552334 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
W20260924 19:32:09.552489 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:09.548212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41485 (local address 127.0.98.62:39053)
0924 19:32:09.548262 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:09.548263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:09.548303 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:32:09.548384 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:09.548386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:09.548398 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:09.548443 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:09.548782 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.549274 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.549951 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.550137 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.550190 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:09.552429 (+ 2239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41485: BlockingRecv error: recv got EOF from 127.0.98.2:41485 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 19:32:09 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:09 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:09.686563 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:09.682435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:50823)
0924 19:32:09.682569 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:09.682579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:09.682637 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:09.682751 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:09.682752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:09.682903 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:09.682947 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:09.682949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.683523 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.683523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:09.684114 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:09.684117 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.684408 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.684409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:09.684445 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:09.684896 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:09.684901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:09.686495 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
W20260924 19:32:09.686640 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:09.682554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50823 (local address 127.0.98.62:39053)
0924 19:32:09.682598 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:09.682600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:09.682624 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:32:09.682684 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:09.682685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:09.682695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:09.682730 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:09.683002 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.683472 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.684181 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:09.684359 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:09.684404 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:09.686578 (+ 2174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50823: BlockingRecv error: recv got EOF from 127.0.98.3:50823 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 19:32:10 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:10 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:10.425612 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:10.421324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:40797)
0924 19:32:10.421455 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:10.421467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:10.421545 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:10.421686 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:10.421687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:10.421844 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:10.421894 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:10.421896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.422519 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.422520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:10.423060 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:10.423062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.423372 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.423373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:10.423408 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:10.423814 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:10.423819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:10.425520 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
W20260924 19:32:10.425654 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:10.421365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40797 (local address 127.0.98.62:39053)
0924 19:32:10.421477 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:10.421479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:10.421526 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:32:10.421606 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:10.421607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:10.421620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:10.421672 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:10.421965 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.422456 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.423133 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.423319 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.423369 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:10.425601 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40797: BlockingRecv error: recv got EOF from 127.0.98.1:40797 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
Sep 24 19:32:10 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:10 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:10.557279 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:10.552871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:50861)
0924 19:32:10.553005 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:10.553016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:10.553071 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:10.553190 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:10.553190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:10.553342 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:10.553387 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:10.553389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.554009 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.554010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:10.554554 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:10.554555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.554847 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.554848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:10.554886 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:10.555320 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:10.555324 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:10.557201 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260924 19:32:10.557353 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:10.552929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50861 (local address 127.0.98.62:39053)
0924 19:32:10.552982 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:10.552984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:10.553069 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:32:10.553125 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:10.553126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:10.553137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:10.553174 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:10.553435 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.553935 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.554615 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.554796 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.554839 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:10.557301 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50861: BlockingRecv error: recv got EOF from 127.0.98.2:50861 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 19:32:10 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:10 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:10.691244 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:10.686982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:52721)
0924 19:32:10.687102 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:10.687115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:10.687189 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:10.687315 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:10.687317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:10.687472 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:10.687517 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:10.687519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.688164 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.688165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:10.688722 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:10.688724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.689037 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.689038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:10.689088 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:10.689511 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:10.689515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:10.691185 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
W20260924 19:32:10.691324 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:10.687073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52721 (local address 127.0.98.62:39053)
0924 19:32:10.687135 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:10.687137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:10.687180 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:32:10.687247 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:10.687248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:10.687261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:10.687305 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:10.687605 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.688092 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.688796 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:10.688981 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:10.689034 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:10.691266 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52721: BlockingRecv error: recv got EOF from 127.0.98.3:52721 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 19:32:11 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:11 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:11.430351 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:11.426103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:47051)
0924 19:32:11.426221 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:11.426231 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:11.426279 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:11.426427 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:11.426428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:11.426606 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:11.426655 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:11.426657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.427372 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.427374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:11.427906 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:11.427908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.428199 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.428200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:11.428235 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:11.428652 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:11.428656 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:11.430294 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
W20260924 19:32:11.430451 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:11.426168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47051 (local address 127.0.98.62:39053)
0924 19:32:11.426291 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:11.426293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:11.426303 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:11.426329 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:11.426330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:11.426342 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:11.426407 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:11.426724 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.427288 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.427967 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.428152 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.428193 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:11.430376 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47051: BlockingRecv error: recv got EOF from 127.0.98.1:47051 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
Sep 24 19:32:11 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:11 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:11.562319 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:11.557763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:58415)
0924 19:32:11.557881 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:11.557901 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:11.557973 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:11.558125 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:11.558126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:11.558299 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:11.558361 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:11.558364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.559060 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.559062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:11.559634 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:11.559637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.560008 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.560010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:11.560067 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:11.560556 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:11.560562 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:11.562242 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
W20260924 19:32:11.562397 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:11.557854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58415 (local address 127.0.98.62:39053)
0924 19:32:11.557937 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:11.557939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:11.557952 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:32:11.558040 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:11.558041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:11.558056 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:11.558107 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:11.558438 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.558974 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.559719 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.559941 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.560003 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:11.562344 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58415: BlockingRecv error: recv got EOF from 127.0.98.2:58415 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 19:32:11 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:11 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:11.696122 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:11.691702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:42137)
0924 19:32:11.691847 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:11.691857 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:11.691920 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:11.692046 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:11.692046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:11.692195 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:11.692242 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:11.692243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.692881 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.692882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:11.693400 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:11.693402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.693689 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.693690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:11.693732 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:11.694151 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:11.694156 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:11.696059 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
W20260924 19:32:11.696204 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:11.691777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42137 (local address 127.0.98.62:39053)
0924 19:32:11.691838 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:11.691840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:11.691915 (+ 75us) server_negotiation.cc:407] Connection header received
0924 19:32:11.691972 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:11.691973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:11.691984 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:11.692022 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:11.692292 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.692823 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.693456 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:11.693638 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:11.693680 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:11.696144 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42137: BlockingRecv error: recv got EOF from 127.0.98.3:42137 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 19:32:12 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:12 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:12.439139 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:12.430799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:59581)
0924 19:32:12.430941 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:12.430954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:12.431014 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:12.431148 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:12.431149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:12.431304 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:12.431351 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:12.431353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.432168 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.432170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:12.432745 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:12.432747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.433074 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.433075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:12.433142 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:12.437359 (+ 4217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:12.437364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:12.439068 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
W20260924 19:32:12.439256 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:12.430913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59581 (local address 127.0.98.62:39053)
0924 19:32:12.431008 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:12.431010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:12.431046 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:32:12.431069 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:12.431070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:12.431081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:12.431136 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:12.431401 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.432095 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.432803 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.432993 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.433071 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:12.439176 (+ 6105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59581: BlockingRecv error: recv got EOF from 127.0.98.1:59581 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":42,"threads_started":1}
Sep 24 19:32:12 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:12 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:12.567066 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:12.562861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:36471)
0924 19:32:12.563011 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:12.563022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:12.563090 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:12.563217 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:12.563217 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:12.563376 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:12.563421 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:12.563423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.564041 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.564042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:12.564598 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:12.564600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.564947 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.564949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:12.565002 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:12.565409 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:12.565413 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:12.566988 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
W20260924 19:32:12.567168 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:12.563010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36471 (local address 127.0.98.62:39053)
0924 19:32:12.563073 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:12.563076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:12.563084 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:32:12.563149 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:12.563150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:12.563161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:12.563206 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:12.563486 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.563968 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.564700 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.564890 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.564943 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:12.567103 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36471: BlockingRecv error: recv got EOF from 127.0.98.2:36471 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 19:32:12 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:12 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:12.701185 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:12.696578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:53447)
0924 19:32:12.696710 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:12.696722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:12.696777 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:12.696894 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:12.696895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:12.697047 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:12.697101 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:12.697103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.697739 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.697740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:12.698340 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:12.698344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.698694 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.698695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:12.698732 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:12.699283 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:12.699288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:12.701113 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
W20260924 19:32:12.701320 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:12.696676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53447 (local address 127.0.98.62:39053)
0924 19:32:12.696771 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:12.696773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:12.696783 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:12.696822 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:12.696822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:12.696834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:12.696880 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:12.697170 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.697681 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.698427 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:12.698639 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:12.698689 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:12.701257 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53447: BlockingRecv error: recv got EOF from 127.0.98.3:53447 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 19:32:13 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:13 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:13.443949 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:13.439554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:60165)
0924 19:32:13.439677 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:13.439687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:13.439732 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:13.439868 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:13.439868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:13.440017 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:13.440067 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:13.440068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.440848 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.440849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:13.441418 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:13.441420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.441786 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.441787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:13.441826 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:13.442277 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:13.442282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:13.443878 (+ 1596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
W20260924 19:32:13.444034 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:13.439615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60165 (local address 127.0.98.62:39053)
0924 19:32:13.439761 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:13.439763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:13.439771 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:32:13.439791 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:13.439791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:13.439801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:13.439855 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:13.440204 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.440765 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.441506 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.441712 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.441767 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:13.443966 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60165: BlockingRecv error: recv got EOF from 127.0.98.1:60165 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
Sep 24 19:32:13 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:13 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:13.571890 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:13.567508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:48379)
0924 19:32:13.567658 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:13.567669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:13.567733 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:13.567871 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:13.567872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:13.568037 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:13.568089 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:13.568090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.568706 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.568708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:13.569297 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:13.569300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.569631 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.569632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:13.569679 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:13.570173 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:13.570178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:13.571830 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
W20260924 19:32:13.572023 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:13.567555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48379 (local address 127.0.98.62:39053)
0924 19:32:13.567611 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:13.567613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:13.567726 (+ 113us) server_negotiation.cc:407] Connection header received
0924 19:32:13.567800 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:13.567800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:13.567812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:13.567859 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:13.568143 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.568651 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.569366 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.569573 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.569628 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:13.571953 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48379: BlockingRecv error: recv got EOF from 127.0.98.2:48379 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 19:32:13 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:13 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:13.706171 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:13.701678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:42813)
0924 19:32:13.701835 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:13.701848 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:13.701921 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:13.702049 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:13.702049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:13.702227 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:13.702274 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:13.702276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.702930 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.702931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:13.703488 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:13.703490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.703843 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.703844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:13.703899 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:13.704329 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:13.704334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:13.706111 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
W20260924 19:32:13.706275 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:13.701791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42813 (local address 127.0.98.62:39053)
0924 19:32:13.701849 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:13.701851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:13.701903 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:32:13.701974 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:13.701974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:13.701988 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:13.702033 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:13.702352 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.702848 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.703574 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:13.703777 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:13.703832 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:13.706211 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42813: BlockingRecv error: recv got EOF from 127.0.98.3:42813 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 19:32:14 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:14 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:14.448387 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:14.444349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:50871)
0924 19:32:14.444456 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:14.444468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:14.444522 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:14.444655 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:14.444656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:14.444811 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:14.444860 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:14.444862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.445483 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.445484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:14.446016 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:14.446018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.446333 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.446334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:14.446372 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:14.446750 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:14.446754 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:14.448328 (+ 1574us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":70,"threads_started":1}
W20260924 19:32:14.448489 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:14.444398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50871 (local address 127.0.98.62:39053)
0924 19:32:14.444498 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:14.444500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:14.444509 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:32:14.444572 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:14.444572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:14.444582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:14.444656 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:14.444913 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.445421 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.446091 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.446281 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.446327 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:14.448415 (+ 2088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50871: BlockingRecv error: recv got EOF from 127.0.98.1:50871 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
Sep 24 19:32:14 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:14 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:14.576807 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:14.572311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:44953)
0924 19:32:14.572431 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:14.572443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:14.572515 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:14.572658 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:14.572658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:14.572823 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:14.572869 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:14.572871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.573509 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.573510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:14.574041 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:14.574043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.574369 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.574369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:14.574422 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:14.574801 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:14.574805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:14.576750 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
W20260924 19:32:14.576902 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:14.572371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44953 (local address 127.0.98.62:39053)
0924 19:32:14.572469 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:14.572471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:14.572496 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:32:14.572566 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:14.572566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:14.572579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:14.572622 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:14.572933 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.573426 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.574110 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.574313 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.574366 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:14.576845 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44953: BlockingRecv error: recv got EOF from 127.0.98.2:44953 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 19:32:14 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:14 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:14.710980 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:14.706666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:42331)
0924 19:32:14.706814 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:14.706827 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:14.706893 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:14.707010 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:14.707010 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:14.707166 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:14.707217 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:14.707218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.707877 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.707878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:14.708411 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:14.708413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.708786 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.708787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:14.708828 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:14.709303 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:14.709308 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:14.710905 (+ 1597us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
W20260924 19:32:14.711081 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:14.706765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42331 (local address 127.0.98.62:39053)
0924 19:32:14.706822 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:14.706824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:14.706876 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:32:14.706937 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:14.706937 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:14.706949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:14.706995 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:14.707273 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.707785 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.708476 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:14.708728 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:14.708780 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:14.711017 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42331: BlockingRecv error: recv got EOF from 127.0.98.3:42331 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 19:32:15 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:15 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:15.452822 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:15.448822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:38865)
0924 19:32:15.448960 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:15.448969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:15.449015 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:15.449175 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:15.449176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:15.449323 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:15.449372 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:15.449374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.449962 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.449963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:15.450477 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:15.450479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.450764 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.450764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:15.450799 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:15.451223 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:15.451228 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:15.452765 (+ 1537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
W20260924 19:32:15.452911 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:15.448929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38865 (local address 127.0.98.62:39053)
0924 19:32:15.449041 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:15.449043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:15.449058 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:32:15.449077 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:15.449078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:15.449087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:15.449145 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:15.449424 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.449911 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.450528 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.450714 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.450759 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:15.452847 (+ 2088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38865: BlockingRecv error: recv got EOF from 127.0.98.1:38865 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
Sep 24 19:32:15 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:15 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:15.581555 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:15.577219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:35049)
0924 19:32:15.577362 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:15.577375 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:15.577453 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:15.577580 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:15.577581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:15.577751 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:15.577796 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:15.577798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.578601 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.578602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:15.579130 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:15.579132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.579441 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.579442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:15.579503 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:15.579896 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:15.579900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:15.581479 (+ 1579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
W20260924 19:32:15.581672 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:15.577284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35049 (local address 127.0.98.62:39053)
0924 19:32:15.577365 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:15.577367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:15.577437 (+ 70us) server_negotiation.cc:407] Connection header received
0924 19:32:15.577516 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:15.577516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:15.577528 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:15.577576 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:15.577863 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.578521 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.579200 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.579386 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.579437 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:15.581574 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35049: BlockingRecv error: recv got EOF from 127.0.98.2:35049 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 19:32:15 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:15 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:15.716051 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:15.711474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:51231)
0924 19:32:15.711636 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:15.711647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:15.711715 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:15.711881 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:15.711882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:15.712068 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:15.712122 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:15.712123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.712816 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.712817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:15.713359 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:15.713361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.713721 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.713723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:15.713784 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:15.714307 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:15.714312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:15.715972 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
W20260924 19:32:15.716140 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:15.711563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51231 (local address 127.0.98.62:39053)
0924 19:32:15.711637 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:15.711639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:15.711708 (+ 69us) server_negotiation.cc:407] Connection header received
0924 19:32:15.711760 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:15.711761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:15.711772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:15.711821 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:15.712189 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.712727 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.713424 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:15.713629 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:15.713689 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:15.716076 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51231: BlockingRecv error: recv got EOF from 127.0.98.3:51231 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 19:32:16 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:16 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:16.457825 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:16.453299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:55589)
0924 19:32:16.453476 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:16.453490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:16.453565 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:16.453706 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:16.453707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:16.453878 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:16.453970 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:16.453972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.454608 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.454609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:16.455178 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:16.455181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.455520 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.455521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:16.455566 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:16.455970 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:16.455974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:16.457764 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":130,"threads_started":1}
W20260924 19:32:16.457933 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:16.453365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55589 (local address 127.0.98.62:39053)
0924 19:32:16.453480 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:16.453482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:16.453546 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:32:16.453618 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:16.453619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:16.453630 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:16.453687 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:16.454028 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.454531 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.455260 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.455453 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.455503 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:16.457857 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55589: BlockingRecv error: recv got EOF from 127.0.98.1:55589 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
Sep 24 19:32:16 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:16 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:16.586601 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:16.582073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:55109)
0924 19:32:16.582218 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:16.582229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:16.582302 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:16.582433 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:16.582433 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:16.582604 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:16.582660 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:16.582662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.583299 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.583300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:16.583844 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:16.583846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.584173 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.584174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:16.584225 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:16.584658 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:16.584662 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:16.586542 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
W20260924 19:32:16.586685 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:16.582135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55109 (local address 127.0.98.62:39053)
0924 19:32:16.582182 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:16.582184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:16.582295 (+ 111us) server_negotiation.cc:407] Connection header received
0924 19:32:16.582362 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:16.582363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:16.582375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:16.582418 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:16.582734 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.583226 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.583920 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.584106 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.584172 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:16.586627 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55109: BlockingRecv error: recv got EOF from 127.0.98.2:55109 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 19:32:16 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:16 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:16.720901 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:16.716561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:49575)
0924 19:32:16.716693 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:16.716705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:16.716769 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:16.716904 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:16.716904 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:16.717058 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:16.717106 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:16.717107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.717736 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.717737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:16.718273 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:16.718274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.718635 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.718636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:16.718682 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:16.719116 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:16.719137 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:16.720838 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":91,"threads_started":1}
W20260924 19:32:16.721082 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:16.716672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49575 (local address 127.0.98.62:39053)
0924 19:32:16.716737 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:16.716739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:16.716752 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:32:16.716837 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:16.716837 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:16.716849 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:16.716892 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:16.717176 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.717665 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.718371 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:16.718567 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:16.718625 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:16.721013 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49575: BlockingRecv error: recv got EOF from 127.0.98.3:49575 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 19:32:17 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:17 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:17.466672 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:17.458309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:57721)
0924 19:32:17.458429 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:17.458443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:17.458515 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:17.458676 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:17.458676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:17.458835 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:17.458882 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:17.458884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.459743 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.459744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:17.460295 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:17.460297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.460718 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.460719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:17.460773 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:17.464965 (+ 4192us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:17.464970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:17.466605 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
W20260924 19:32:17.466773 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:17.458420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57721 (local address 127.0.98.62:39053)
0924 19:32:17.458545 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:17.458547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:17.458556 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:32:17.458577 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:17.458578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:17.458587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:17.458645 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:17.458964 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.459654 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.460390 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.460589 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.460695 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:17.466702 (+ 6007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57721: BlockingRecv error: recv got EOF from 127.0.98.1:57721 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
Sep 24 19:32:17 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:17 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:17.591702 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:17.587131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:42311)
0924 19:32:17.587274 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:17.587284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:17.587347 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:17.587487 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:17.587488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:17.587659 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:17.587707 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:17.587709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.588429 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.588430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:17.589106 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:17.589109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.589575 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.589576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:17.589638 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:17.590086 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:17.590092 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:17.591643 (+ 1551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260924 19:32:17.591794 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:17.587246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42311 (local address 127.0.98.62:39053)
0924 19:32:17.587296 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:17.587298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:17.587332 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:32:17.587406 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:17.587406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:17.587420 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:17.587471 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:17.587784 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.588334 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.589190 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.589505 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.589568 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:17.591736 (+ 2168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42311: BlockingRecv error: recv got EOF from 127.0.98.2:42311 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 19:32:17 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:17 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:17.725430 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:17.721300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:37081)
0924 19:32:17.721434 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:17.721444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:17.721488 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:17.721618 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:17.721619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:17.721787 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:17.721833 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:17.721835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.722436 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.722437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:17.722958 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:17.722960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.723232 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.723233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:17.723269 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:17.723668 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:17.723673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:17.725370 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
W20260924 19:32:17.725525 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:17.721420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37081 (local address 127.0.98.62:39053)
0924 19:32:17.721465 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:17.721467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:17.721477 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:17.721556 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:17.721557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:17.721568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:17.721605 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:17.721887 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.722379 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.723015 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:17.723187 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:17.723229 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:17.725462 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37081: BlockingRecv error: recv got EOF from 127.0.98.3:37081 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 19:32:18 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:18 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:18.471655 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:18.467178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:37155)
0924 19:32:18.467341 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:18.467352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:18.467408 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:18.467525 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:18.467526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:18.467692 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:18.467741 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:18.467743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.468409 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.468410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:18.469008 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:18.469010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.469461 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.469463 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:18.469524 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:18.469957 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:18.469961 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:18.471579 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
W20260924 19:32:18.471736 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:18.467302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37155 (local address 127.0.98.62:39053)
0924 19:32:18.467408 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:18.467410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:18.467419 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:32:18.467444 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:18.467444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:18.467454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:18.467507 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:18.467814 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.468323 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.469136 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.469367 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.469443 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:18.471675 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37155: BlockingRecv error: recv got EOF from 127.0.98.1:37155 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":45,"threads_started":1}
Sep 24 19:32:18 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:18 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:18.596701 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:18.592243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:60879)
0924 19:32:18.592391 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:18.592404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:18.592482 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:18.592620 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:18.592621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:18.592838 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:18.592902 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:18.592905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.593548 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.593549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:18.594082 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:18.594084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.594421 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.594422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:18.594499 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:18.594901 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:18.594905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:18.596589 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
W20260924 19:32:18.596764 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:18.592305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60879 (local address 127.0.98.62:39053)
0924 19:32:18.592371 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:18.592373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:18.592483 (+ 110us) server_negotiation.cc:407] Connection header received
0924 19:32:18.592532 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:18.592533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:18.592545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:18.592604 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:18.592989 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.593484 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.594144 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.594341 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.594400 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:18.596712 (+ 2312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60879: BlockingRecv error: recv got EOF from 127.0.98.2:60879 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 19:32:18 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:18 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:18.730746 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:18.725939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:36189)
0924 19:32:18.726066 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:18.726078 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:18.726143 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:18.726269 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:18.726269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:18.726440 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:18.726485 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:18.726487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.727306 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.727307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:18.727984 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:18.727987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.728326 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.728327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:18.728392 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:18.728860 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:18.728865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:18.730685 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
W20260924 19:32:18.730866 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:18.726028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36189 (local address 127.0.98.62:39053)
0924 19:32:18.726077 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:18.726079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:18.726127 (+ 48us) server_negotiation.cc:407] Connection header received
0924 19:32:18.726193 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:18.726193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:18.726208 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:18.726252 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:18.726558 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.727238 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.728060 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:18.728267 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:18.728321 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:18.730800 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36189: BlockingRecv error: recv got EOF from 127.0.98.3:36189 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 19:32:19 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:19 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:19.476898 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:19.472159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:59387)
0924 19:32:19.472297 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:19.472307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:19.472363 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:19.472495 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:19.472495 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:19.472693 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:19.472744 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:19.472746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.473488 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.473489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:19.474022 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:19.474024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.474373 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.474374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:19.474413 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:19.474951 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:19.474957 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:19.476839 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
W20260924 19:32:19.477005 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:19.472261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59387 (local address 127.0.98.62:39053)
0924 19:32:19.472386 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:19.472387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:19.472397 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:19.472418 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:19.472419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:19.472427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:19.472481 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:19.472849 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.473398 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.474107 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.474309 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.474371 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:19.476936 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59387: BlockingRecv error: recv got EOF from 127.0.98.1:59387 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
Sep 24 19:32:19 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:19 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:19.601372 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:19.597083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:49427)
0924 19:32:19.597215 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:19.597226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:19.597292 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:19.597435 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:19.597436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:19.597603 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:19.597649 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:19.597651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.598283 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.598284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:19.598825 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:19.598827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.599139 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.599140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:19.599192 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:19.599593 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:19.599597 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:19.601303 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
W20260924 19:32:19.601462 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:19.597149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49427 (local address 127.0.98.62:39053)
0924 19:32:19.597196 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:19.597198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:19.597285 (+ 87us) server_negotiation.cc:407] Connection header received
0924 19:32:19.597361 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:19.597362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:19.597372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:19.597413 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:19.597719 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.598210 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.598892 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.599079 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.599133 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:19.601402 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49427: BlockingRecv error: recv got EOF from 127.0.98.2:49427 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 19:32:19 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:19 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:19.735947 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:19.731253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:44271)
0924 19:32:19.731388 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:19.731401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:19.731468 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:19.731598 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:19.731599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:19.731760 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:19.731806 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:19.731807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.732467 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.732468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:19.733022 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:19.733024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.733412 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.733413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:19.733479 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:19.733888 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:19.733892 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:19.735857 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260924 19:32:19.735992 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:19.731362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44271 (local address 127.0.98.62:39053)
0924 19:32:19.731407 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:19.731409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:19.731451 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:32:19.731522 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:19.731522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:19.731535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:19.731579 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:19.731854 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.732393 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.733102 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:19.733294 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:19.733354 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:19.735940 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44271: BlockingRecv error: recv got EOF from 127.0.98.3:44271 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 19:32:20 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:20 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:20.486846 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:20.477405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:44327)
0924 19:32:20.477545 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:20.477556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:20.477608 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:20.477736 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:20.477737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:20.477887 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:20.477937 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:20.477939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.478552 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.478553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:20.479109 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:20.479111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.479457 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.479458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:20.479498 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:20.484863 (+ 5365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:20.484870 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:20.486756 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
W20260924 19:32:20.486929 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:20.477509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44327 (local address 127.0.98.62:39053)
0924 19:32:20.477622 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:20.477624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:20.477632 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:32:20.477659 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:20.477659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:20.477668 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:20.477724 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:20.478001 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.478496 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.479200 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.479396 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.479451 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:20.486868 (+ 7417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44327: BlockingRecv error: recv got EOF from 127.0.98.1:44327 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
Sep 24 19:32:20 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:20 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:20.607296 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:20.601825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:41085)
0924 19:32:20.602102 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:20.602116 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:20.602202 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:20.602396 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:20.602397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:20.602567 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:20.602614 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:20.602616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.603563 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.603565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:20.604137 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:20.604141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.604607 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.604608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:20.604697 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:20.605168 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:20.605173 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:20.607235 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":53,"threads_started":1}
W20260924 19:32:20.607512 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:20.601918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41085 (local address 127.0.98.62:39053)
0924 19:32:20.601990 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:20.602146 (+ 156us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:20.602184 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:32:20.602289 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:20.602290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:20.602303 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:20.602346 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:20.602717 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.603471 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.604230 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.604528 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.604599 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:20.607445 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41085: BlockingRecv error: recv got EOF from 127.0.98.2:41085 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 19:32:20 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:20 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:20.740876 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:20.736392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:39953)
0924 19:32:20.736516 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:20.736528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:20.736601 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:20.736772 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:20.736773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:20.736953 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:20.737000 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:20.737002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.737715 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.737717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:20.738246 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:20.738248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.738563 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.738564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:20.738621 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:20.739053 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:20.739057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:20.740800 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
W20260924 19:32:20.740952 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:20.736489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39953 (local address 127.0.98.62:39053)
0924 19:32:20.736551 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:20.736553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:20.736578 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:32:20.736691 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:20.736692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:20.736707 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:20.736755 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:20.737074 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.737638 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.738323 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:20.738504 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:20.738555 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:20.740899 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39953: BlockingRecv error: recv got EOF from 127.0.98.3:39953 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 19:32:21 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:21 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:21.491916 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:21.487402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:55153)
0924 19:32:21.487553 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:21.487565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:21.487624 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:21.487781 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:21.487782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:21.487934 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:21.487982 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:21.487983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.488683 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.488684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:21.489346 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:21.489350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.489756 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.489757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:21.489810 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:21.490228 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:21.490232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:21.491840 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":57,"threads_started":1}
W20260924 19:32:21.492007 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:21.487563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55153 (local address 127.0.98.62:39053)
0924 19:32:21.487665 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:21.487666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:21.487675 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:32:21.487698 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:21.487698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:21.487707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:21.487763 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:21.488057 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.488567 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.489450 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.489694 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.489746 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:21.491945 (+ 2199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55153: BlockingRecv error: recv got EOF from 127.0.98.1:55153 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":44,"threads_started":1}
Sep 24 19:32:21 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:21 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:21.613336 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:21.607795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:46949)
0924 19:32:21.607905 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:21.607917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:21.607983 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:21.608143 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:21.608144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:21.608413 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:21.608460 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:21.608462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.609451 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.609452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:21.610235 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:21.610237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.610706 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.610707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:21.610759 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:21.611251 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:21.611256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:21.613238 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
W20260924 19:32:21.613408 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:21.607839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46949 (local address 127.0.98.62:39053)
0924 19:32:21.607940 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:21.607941 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:21.607968 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:32:21.608046 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:21.608047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:21.608062 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:21.608115 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:21.608535 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.609377 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.610324 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.610633 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.610703 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:21.613348 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46949: BlockingRecv error: recv got EOF from 127.0.98.2:46949 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 19:32:21 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:21 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:21.745862 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:21.741352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.3:51493)
0924 19:32:21.741500 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:21.741510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:21.741565 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:21.741689 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:21.741690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:21.741853 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:21.741900 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:21.741902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.742521 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.742523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:21.743064 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:21.743066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.743391 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.743392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:21.743456 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:21.743940 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:21.743945 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:21.745787 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
W20260924 19:32:21.745944 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:21.741428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51493 (local address 127.0.98.62:39053)
0924 19:32:21.741479 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:21.741481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:21.741562 (+ 81us) server_negotiation.cc:407] Connection header received
0924 19:32:21.741624 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:21.741625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:21.741636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:21.741675 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:21.741949 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.742446 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.743126 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:21.743322 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:21.743368 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:21.745882 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51493: BlockingRecv error: recv got EOF from 127.0.98.3:51493 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 19:32:22 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:22 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:22.500836 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:22.492440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.1:47543)
0924 19:32:22.492568 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:22.492580 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:22.492625 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:22.492787 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:22.492787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:22.492942 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:22.492992 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:22.492994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.493607 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.493608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:22.494160 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:22.494162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.494468 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.494468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:22.494508 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:22.499156 (+ 4648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:22.499160 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:22.500778 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
W20260924 19:32:22.500945 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:22.492528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47543 (local address 127.0.98.62:39053)
0924 19:32:22.492648 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:22.492650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:22.492688 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:32:22.492716 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:22.492717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:22.492727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:22.492775 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:22.493043 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.493537 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.494227 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.494413 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.494460 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:22.500868 (+ 6408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47543: BlockingRecv error: recv got EOF from 127.0.98.1:47543 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
Sep 24 19:32:22 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:32:22 dist-test-slave-m8jb krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:32:22.617872 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:22.613819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39053 (local address 127.0.98.2:34161)
0924 19:32:22.613923 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:22.613935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:22.613995 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:22.614127 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:22.614128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:22.614313 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:22.614361 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:22.614363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.614965 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.614965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:22.615501 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:22.615502 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.615789 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.615790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:22.615832 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:22.616200 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:22.616204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:22.617807 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39053: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
W20260924 19:32:22.617952 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:22.613856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34161 (local address 127.0.98.62:39053)
0924 19:32:22.613903 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:22.613905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:22.613994 (+ 89us) server_negotiation.cc:407] Connection header received
0924 19:32:22.614047 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:22.614047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:22.614060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:22.614114 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:22.614417 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.614902 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.615559 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.615735 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.615778 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:22.617892 (+ 2114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34161: BlockingRecv error: recv got EOF from 127.0.98.2:34161 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:32:22.677449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 9109
I20260924 19:32:22.681422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 9261
I20260924 19:32:22.685231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 9394
I20260924 19:32:22.688665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 9039
2026-09-24T19:32:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17285 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T19:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:32:22Z Disabled control of system clock
I20260924 19:32:22.713672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36583
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:32:22.791169 9590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:22.791380 9590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:22.791399 9590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:22.792502 9590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:32:22.792536 9590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:22.792547 9590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:22.792558 9590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:32:22.792569 9590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:32:22.793893 9590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:22.794080 9590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:22.794312 9590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:22.799202 9598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:22.799295 9596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:22.799227 9595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:22.799302 9590 server_base.cc:1034] running on GCE node
I20260924 19:32:22.799659 9590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:22.799938 9590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:22.801100 9590 hybrid_clock.cc:648] HybridClock initialized: now 1790278342801084 us; error 31 us; skew 500 ppm
I20260924 19:32:22.802367 9590 webserver.cc:460] Webserver started at http://127.0.98.62:38265/ using document root <none> and password file <none>
I20260924 19:32:22.802619 9590 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:22.802670 9590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:22.802809 9590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:22.803906 9590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "199bfc51a0e4475386fedc9050004663"
format_stamp: "Formatted at 2026-09-24 19:32:22 on dist-test-slave-m8jb"
server_key: "126f0060aea8e4b49fecbc07274cf22f"
server_key_iv: "6ffe0b14054246c000518aafd3632fef"
server_key_version: "encryptionkey@0"
I20260924 19:32:22.804279 9590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "199bfc51a0e4475386fedc9050004663"
format_stamp: "Formatted at 2026-09-24 19:32:22 on dist-test-slave-m8jb"
server_key: "126f0060aea8e4b49fecbc07274cf22f"
server_key_iv: "6ffe0b14054246c000518aafd3632fef"
server_key_version: "encryptionkey@0"
I20260924 19:32:22.805670 9590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:22.806480 9604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:22.806619 9590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:22.806699 9590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "199bfc51a0e4475386fedc9050004663"
format_stamp: "Formatted at 2026-09-24 19:32:22 on dist-test-slave-m8jb"
server_key: "126f0060aea8e4b49fecbc07274cf22f"
server_key_iv: "6ffe0b14054246c000518aafd3632fef"
server_key_version: "encryptionkey@0"
I20260924 19:32:22.806766 9590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:22.822096 9590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:22.822391 9590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:22.822531 9590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:22.826184 9590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36583
I20260924 19:32:22.826241 9656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36583 every 8 connection(s)
I20260924 19:32:22.826556 9590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:32:22.827212 9657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:22.827555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 9590
I20260924 19:32:22.827632 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:32:22.827827 392 external_mini_cluster.cc:1428] Setting key 38452a4a8482ce9eb5c6962d0d66d805
I20260924 19:32:22.830906 9657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663: Bootstrap starting.
I20260924 19:32:22.831491 9657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:22.831748 9657 log.cc:824] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:22.832353 9657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663: No bootstrap required, opened a new log
I20260924 19:32:22.833418 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:22.828339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.0.98.62:36583)
0924 19:32:22.828556 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:22.828566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:22.828618 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:32:22.829100 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:22.829109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:22.829206 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:22.829363 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:22.829531 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.830834 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.831633 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.832087 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.832206 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:22.832853 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:22.832857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:22.832857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:32:22.833019 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:22.833050 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:32:22.833061 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:22.833113 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:22.833183 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 19:32:22.833261 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260924 19:32:22.834275 9657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199bfc51a0e4475386fedc9050004663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36583 } }
I20260924 19:32:22.834410 9657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:22.834440 9657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 199bfc51a0e4475386fedc9050004663, State: Initialized, Role: FOLLOWER
I20260924 19:32:22.834555 9657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199bfc51a0e4475386fedc9050004663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36583 } }
I20260924 19:32:22.834623 9657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:32:22.834663 9657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:32:22.834717 9657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:22.835237 9657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199bfc51a0e4475386fedc9050004663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36583 } }
I20260924 19:32:22.835338 9657 leader_election.cc:302] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 199bfc51a0e4475386fedc9050004663; no voters:
I20260924 19:32:22.835512 9657 leader_election.cc:288] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:32:22.835546 9662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:22.835678 9662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [term 1 LEADER]: Becoming Leader. State: Replica: 199bfc51a0e4475386fedc9050004663, State: Running, Role: LEADER
I20260924 19:32:22.835758 9657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:32:22.835817 9662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199bfc51a0e4475386fedc9050004663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36583 } }
I20260924 19:32:22.836143 9663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "199bfc51a0e4475386fedc9050004663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199bfc51a0e4475386fedc9050004663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36583 } } }
I20260924 19:32:22.836247 9663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:22.836155 9664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 199bfc51a0e4475386fedc9050004663. Latest consensus state: current_term: 1 leader_uuid: "199bfc51a0e4475386fedc9050004663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "199bfc51a0e4475386fedc9050004663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36583 } } }
I20260924 19:32:22.836359 9664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:22.836771 9671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:32:22.837276 9671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:32:22.838745 9671 catalog_manager.cc:1347] Generated new cluster ID: 59d6c50d1d4d47e2a6078bc247ea08e0
I20260924 19:32:22.838786 9671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:32:22.853684 9671 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:32:22.854208 9671 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:32:22.864007 9671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 199bfc51a0e4475386fedc9050004663: Generated new TSK 0
I20260924 19:32:22.864197 9671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:32:22.866721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36583
--builtin_ntp_servers=127.0.98.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:32:22.947521 9681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:22.947731 9681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:22.947751 9681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:22.948927 9681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:22.948967 9681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:22.948992 9681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:32:22.950302 9681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:22.950524 9681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:22.950809 9681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:22.955482 9689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:22.955488 9686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:22.955616 9681 server_base.cc:1034] running on GCE node
W20260924 19:32:22.955598 9687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:22.955967 9681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:22.956203 9681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:22.957342 9681 hybrid_clock.cc:648] HybridClock initialized: now 1790278342957325 us; error 34 us; skew 500 ppm
I20260924 19:32:22.958491 9681 webserver.cc:460] Webserver started at http://127.0.98.1:36793/ using document root <none> and password file <none>
I20260924 19:32:22.958719 9681 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:22.958779 9681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:22.958904 9681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:22.959916 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "bcb82baa1bce43b2bb8f2561e1d1a645"
format_stamp: "Formatted at 2026-09-24 19:32:22 on dist-test-slave-m8jb"
server_key: "dd2a3faa39bb4a01289fe1ea628fa2fa"
server_key_iv: "c55d223d1ec6436d7973ff3c87c9ac17"
server_key_version: "encryptionkey@0"
I20260924 19:32:22.960244 9681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bcb82baa1bce43b2bb8f2561e1d1a645"
format_stamp: "Formatted at 2026-09-24 19:32:22 on dist-test-slave-m8jb"
server_key: "dd2a3faa39bb4a01289fe1ea628fa2fa"
server_key_iv: "c55d223d1ec6436d7973ff3c87c9ac17"
server_key_version: "encryptionkey@0"
I20260924 19:32:22.961476 9681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:32:22.962097 9695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:22.962258 9681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:32:22.962337 9681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "bcb82baa1bce43b2bb8f2561e1d1a645"
format_stamp: "Formatted at 2026-09-24 19:32:22 on dist-test-slave-m8jb"
server_key: "dd2a3faa39bb4a01289fe1ea628fa2fa"
server_key_iv: "c55d223d1ec6436d7973ff3c87c9ac17"
server_key_version: "encryptionkey@0"
I20260924 19:32:22.962400 9681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:22.976528 9681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:22.976902 9681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:22.977037 9681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:22.977257 9681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:22.977573 9681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:22.977605 9681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:22.977633 9681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:22.977658 9681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:22.983183 9681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45711
I20260924 19:32:22.983242 9808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45711 every 8 connection(s)
I20260924 19:32:22.983562 9681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:32:22.988416 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:22.984278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41535 (local address 127.0.98.62:36583)
0924 19:32:22.984339 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:22.984341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:22.984518 (+ 177us) server_negotiation.cc:407] Connection header received
0924 19:32:22.985049 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:22.985050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:22.985063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:22.985140 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:22.985763 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.986265 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.987195 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.987408 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.987469 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:22.988187 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:22.988188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:22.988188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:32:22.988193 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:22.988208 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:32:22.988211 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:22.988251 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:22.988330 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:32:22.988354 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:32:22.988597 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:22.984212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36583 (local address 127.0.98.1:41535)
0924 19:32:22.984389 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:22.984431 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:22.984948 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:22.985203 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:22.985206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:22.985215 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:22.985689 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:22.985693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.986333 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.986333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:22.987099 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:22.987100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:22.987480 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:22.987481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:22.987594 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:22.988056 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:32:22.988057 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:22.988087 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:32:22.988111 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:22.988261 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:22.988264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:22.988266 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:32:22.988315 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:32:22.988397 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260924 19:32:22.989065 9809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36583
I20260924 19:32:22.989161 9809 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:22.989347 9809 heartbeater.cc:511] Master 127.0.98.62:36583 requested a full tablet report, sending...
I20260924 19:32:22.989809 9621 ts_manager.cc:194] Registered new tserver with Master: bcb82baa1bce43b2bb8f2561e1d1a645 (127.0.98.1:45711)
I20260924 19:32:22.990382 9621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41535
I20260924 19:32:22.991320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 9681
I20260924 19:32:22.991396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:32:22.991556 392 external_mini_cluster.cc:1428] Setting key f70015801391602b02b5cbc048a588d0
I20260924 19:32:22.992326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36583
--builtin_ntp_servers=127.0.98.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:32:23.070972 9812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:23.071190 9812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:23.071210 9812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:23.072366 9812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:23.072407 9812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:23.072432 9812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:32:23.073879 9812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:23.074112 9812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:23.074388 9812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:32:23.079213 9812 server_base.cc:1034] running on GCE node
W20260924 19:32:23.079159 9817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.079175 9818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.079181 9820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.079571 9812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:23.079794 9812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:23.080943 9812 hybrid_clock.cc:648] HybridClock initialized: now 1790278343080927 us; error 28 us; skew 500 ppm
I20260924 19:32:23.082072 9812 webserver.cc:460] Webserver started at http://127.0.98.2:41343/ using document root <none> and password file <none>
I20260924 19:32:23.082293 9812 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:23.082340 9812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:23.082472 9812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:23.083469 9812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "c39cdb03057b43d2a448c20f2d587332"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "0d85407ac0acfa098cb05c35a9025c64"
server_key_iv: "895f677efde097b83e82448944cec0f1"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.083797 9812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c39cdb03057b43d2a448c20f2d587332"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "0d85407ac0acfa098cb05c35a9025c64"
server_key_iv: "895f677efde097b83e82448944cec0f1"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.084977 9812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:23.085641 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.085783 9812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:23.085861 9812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "c39cdb03057b43d2a448c20f2d587332"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "0d85407ac0acfa098cb05c35a9025c64"
server_key_iv: "895f677efde097b83e82448944cec0f1"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.085925 9812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:23.095010 9812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:23.095252 9812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:23.095348 9812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:23.095502 9812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:23.095773 9812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:23.095808 9812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.095832 9812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:23.095849 9812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.101112 9812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44411
I20260924 19:32:23.101187 9939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44411 every 8 connection(s)
I20260924 19:32:23.101455 9812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:32:23.106217 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.102099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53927 (local address 127.0.98.62:36583)
0924 19:32:23.102140 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:23.102142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:23.102368 (+ 226us) server_negotiation.cc:407] Connection header received
0924 19:32:23.102837 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:23.102838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:23.102847 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:23.102905 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:23.103559 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:23.104058 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:23.105081 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:23.105263 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:23.105312 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:23.105984 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:23.105985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:23.105985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:32:23.105989 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:23.106004 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:32:23.106008 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:23.106081 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:23.106137 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 19:32:23.106157 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:32:23.106416 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.102074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36583 (local address 127.0.98.2:53927)
0924 19:32:23.102263 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:23.102300 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:23.102758 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:23.102969 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:23.102972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:23.102981 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:23.103475 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:23.103479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:23.104127 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:23.104127 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:23.104984 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:23.104986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:23.105325 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:23.105326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:23.105427 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:23.105869 (+ 442us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:32:23.105870 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:23.105897 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:32:23.105920 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:23.106084 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:23.106086 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:23.106089 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:23.106133 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:32:23.106215 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260924 19:32:23.106563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 9812
I20260924 19:32:23.106647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:32:23.106837 392 external_mini_cluster.cc:1428] Setting key 27af6a50ea86d023a69a761f8328764e
I20260924 19:32:23.106899 9940 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36583
I20260924 19:32:23.106992 9940 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:23.107188 9940 heartbeater.cc:511] Master 127.0.98.62:36583 requested a full tablet report, sending...
I20260924 19:32:23.107546 9621 ts_manager.cc:194] Registered new tserver with Master: c39cdb03057b43d2a448c20f2d587332 (127.0.98.2:44411)
I20260924 19:32:23.107775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36583
--builtin_ntp_servers=127.0.98.20:36265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:32:23.108021 9621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53927
W20260924 19:32:23.186820 9943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:23.187022 9943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:23.187041 9943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:23.188181 9943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:23.188230 9943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:23.188263 9943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:32:23.189692 9943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:23.189922 9943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:23.190196 9943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:23.194852 9949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.194852 9951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.194888 9948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.195007 9943 server_base.cc:1034] running on GCE node
I20260924 19:32:23.195194 9943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:23.195406 9943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:23.196532 9943 hybrid_clock.cc:648] HybridClock initialized: now 1790278343196514 us; error 35 us; skew 500 ppm
I20260924 19:32:23.197739 9943 webserver.cc:460] Webserver started at http://127.0.98.3:39235/ using document root <none> and password file <none>
I20260924 19:32:23.197949 9943 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:23.197993 9943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:23.198110 9943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:23.199148 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee11f490dd794fa4a01d3c5c69b3c682"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "832047464a56dc68748d5b15a802d09f"
server_key_iv: "2bb88d145146997e426e96562651266b"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.199465 9943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee11f490dd794fa4a01d3c5c69b3c682"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "832047464a56dc68748d5b15a802d09f"
server_key_iv: "2bb88d145146997e426e96562651266b"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.200729 9943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:23.201434 9957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.201609 9943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:23.201680 9943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "ee11f490dd794fa4a01d3c5c69b3c682"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "832047464a56dc68748d5b15a802d09f"
server_key_iv: "2bb88d145146997e426e96562651266b"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.201751 9943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:23.212756 9943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:23.213028 9943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:23.213147 9943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:23.213349 9943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:23.213650 9943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:23.213683 9943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.213713 9943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:23.213737 9943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.219060 9943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45383
I20260924 19:32:23.219113 10070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45383 every 8 connection(s)
I20260924 19:32:23.219396 9943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:32:23.222231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 9943
I20260924 19:32:23.222325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:32:23.222532 392 external_mini_cluster.cc:1428] Setting key a90a6d6c607cf6425ea7713f8228fab5
I20260924 19:32:23.224514 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.220035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39579 (local address 127.0.98.62:36583)
0924 19:32:23.220074 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:23.220076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:23.220292 (+ 216us) server_negotiation.cc:407] Connection header received
0924 19:32:23.220789 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:23.220790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:23.220800 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:23.220845 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:23.221458 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:23.221932 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:23.223287 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:23.223508 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:23.223557 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:23.224306 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:23.224307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:23.224308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:32:23.224312 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:23.224326 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:32:23.224330 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:23.224368 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:23.224435 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:32:23.224458 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:32:23.224761 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.220022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36583 (local address 127.0.98.3:39579)
0924 19:32:23.220176 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:23.220207 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:23.220705 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:23.220906 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:23.220910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:23.220919 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:23.221402 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:23.221406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:23.222039 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:23.222040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:23.223212 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:23.223215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:23.223574 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:23.223575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:23.223686 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:23.224171 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:32:23.224171 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:23.224205 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:32:23.224228 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:23.224383 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:23.224387 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:23.224389 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:32:23.224438 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:32:23.224520 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
I20260924 19:32:23.225189 10071 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36583
I20260924 19:32:23.225270 10071 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:23.225438 10071 heartbeater.cc:511] Master 127.0.98.62:36583 requested a full tablet report, sending...
I20260924 19:32:23.225776 9621 ts_manager.cc:194] Registered new tserver with Master: ee11f490dd794fa4a01d3c5c69b3c682 (127.0.98.3:45383)
I20260924 19:32:23.226097 9621 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39579
I20260924 19:32:23.234153 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:32:23.235880 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:23.235419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.0.98.62:36583)
0924 19:32:23.235459 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:23.235462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:23.235554 (+ 92us) server_negotiation.cc:407] Connection header received
0924 19:32:23.235631 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:23.235632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:23.235642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:23.235689 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:23.235815 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51240: BlockingRecv error: recv got EOF from 127.0.0.1:51240 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:32:23.235934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 9681
I20260924 19:32:23.239871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 9812
I20260924 19:32:23.243664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 9943
I20260924 19:32:23.247290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 9590
2026-09-24T19:32:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (557 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T19:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:32:23Z Disabled control of system clock
I20260924 19:32:23.271134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33407
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:32:23.349381 10082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:23.349578 10082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:23.349597 10082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:23.350687 10082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:32:23.350723 10082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:23.350733 10082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:23.350744 10082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:32:23.350754 10082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:32:23.352013 10082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:23.352210 10082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:23.352447 10082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:23.357262 10090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.357296 10088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.357316 10082 server_base.cc:1034] running on GCE node
W20260924 19:32:23.357303 10087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.357618 10082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:23.357878 10082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:23.359017 10082 hybrid_clock.cc:648] HybridClock initialized: now 1790278343359006 us; error 26 us; skew 500 ppm
I20260924 19:32:23.360203 10082 webserver.cc:460] Webserver started at http://127.0.98.62:44579/ using document root <none> and password file <none>
I20260924 19:32:23.360445 10082 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:23.360491 10082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:23.360608 10082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:23.361683 10082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "059ef913d2de423ab81ce56e8e2c90a8"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "9f89f19b9b17a4fdcda7d2b51a65e253"
server_key_iv: "c849618fd42d5dafd97c1d79b4d8ccf3"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.361995 10082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "059ef913d2de423ab81ce56e8e2c90a8"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "9f89f19b9b17a4fdcda7d2b51a65e253"
server_key_iv: "c849618fd42d5dafd97c1d79b4d8ccf3"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.363184 10082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:32:23.363793 10096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.363924 10082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:32:23.364002 10082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "059ef913d2de423ab81ce56e8e2c90a8"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "9f89f19b9b17a4fdcda7d2b51a65e253"
server_key_iv: "c849618fd42d5dafd97c1d79b4d8ccf3"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.364061 10082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:23.367251 10082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:23.367493 10082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:23.367614 10082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:23.371263 10082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33407
I20260924 19:32:23.371292 10148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33407 every 8 connection(s)
I20260924 19:32:23.371592 10082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:32:23.372169 10149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:23.374882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 10082
I20260924 19:32:23.374939 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8: Bootstrap starting.
I20260924 19:32:23.374982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:32:23.375169 392 external_mini_cluster.cc:1428] Setting key b5a3dbb1b13d8ed7e78df89f304fc879
I20260924 19:32:23.375625 10149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:23.375963 10149 log.cc:824] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:23.376741 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8: No bootstrap required, opened a new log
I20260924 19:32:23.377635 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.375679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.0.98.62:33407)
0924 19:32:23.375868 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:23.375875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:23.375899 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:32:23.376383 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:23.376392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:23.376457 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:23.376576 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:23.377213 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:23.377216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:23.377216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:32:23.377220 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:23.377260 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:32:23.377270 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:23.377334 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:23.377402 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 19:32:23.377453 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260924 19:32:23.378638 10149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ef913d2de423ab81ce56e8e2c90a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } }
I20260924 19:32:23.378798 10149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:23.378821 10149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059ef913d2de423ab81ce56e8e2c90a8, State: Initialized, Role: FOLLOWER
I20260924 19:32:23.378913 10149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ef913d2de423ab81ce56e8e2c90a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } }
I20260924 19:32:23.378989 10149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:32:23.379016 10149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:32:23.379047 10149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:23.379827 10149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ef913d2de423ab81ce56e8e2c90a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } }
I20260924 19:32:23.379909 10149 leader_election.cc:302] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 059ef913d2de423ab81ce56e8e2c90a8; no voters:
I20260924 19:32:23.380025 10149 leader_election.cc:288] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:32:23.380074 10154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:23.380224 10154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [term 1 LEADER]: Becoming Leader. State: Replica: 059ef913d2de423ab81ce56e8e2c90a8, State: Running, Role: LEADER
I20260924 19:32:23.380265 10149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:32:23.380308 10154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ef913d2de423ab81ce56e8e2c90a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } }
I20260924 19:32:23.380833 10156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059ef913d2de423ab81ce56e8e2c90a8. Latest consensus state: current_term: 1 leader_uuid: "059ef913d2de423ab81ce56e8e2c90a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ef913d2de423ab81ce56e8e2c90a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } }
I20260924 19:32:23.380923 10156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:23.381296 10155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "059ef913d2de423ab81ce56e8e2c90a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059ef913d2de423ab81ce56e8e2c90a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33407 } } }
I20260924 19:32:23.381356 10155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:23.381434 10163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:32:23.381867 10163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:32:23.383278 10163 catalog_manager.cc:1347] Generated new cluster ID: 727cdc8baa4942b2bcf62cd3b8a1df26
I20260924 19:32:23.383327 10163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:32:23.401500 10163 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:32:23.401678 10163 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:32:23.414355 10163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 059ef913d2de423ab81ce56e8e2c90a8: Generated new TSK 0
I20260924 19:32:23.414666 10163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:32:23.427146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33407
--builtin_ntp_servers=127.0.98.20:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:32:23.505602 10173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:23.505802 10173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:23.505822 10173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:23.506959 10173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:23.506996 10173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:23.507021 10173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:32:23.508351 10173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:23.508566 10173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:23.508863 10173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:23.513494 10181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.513520 10179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.513518 10173 server_base.cc:1034] running on GCE node
W20260924 19:32:23.513502 10178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.513818 10173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:23.514021 10173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:23.515173 10173 hybrid_clock.cc:648] HybridClock initialized: now 1790278343515159 us; error 29 us; skew 500 ppm
I20260924 19:32:23.516369 10173 webserver.cc:460] Webserver started at http://127.0.98.1:46413/ using document root <none> and password file <none>
I20260924 19:32:23.516590 10173 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:23.516670 10173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:23.516793 10173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:23.517846 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb699238f6dc4648b60e37c2e6353efe"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "e0e04e7ef91e9b1c1d8a90229134fbeb"
server_key_iv: "edf07a380118d43c031513de62f53559"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.518177 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb699238f6dc4648b60e37c2e6353efe"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "e0e04e7ef91e9b1c1d8a90229134fbeb"
server_key_iv: "edf07a380118d43c031513de62f53559"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.519447 10173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:23.520190 10187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.520347 10173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:23.520430 10173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "bb699238f6dc4648b60e37c2e6353efe"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "e0e04e7ef91e9b1c1d8a90229134fbeb"
server_key_iv: "edf07a380118d43c031513de62f53559"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.520483 10173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:23.523737 10173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:23.523957 10173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:23.524048 10173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:23.524187 10173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:23.524418 10173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:23.524451 10173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.524469 10173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:23.524487 10173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.529744 10173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42221
I20260924 19:32:23.529812 10300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42221 every 8 connection(s)
I20260924 19:32:23.530102 10173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:32:23.531803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 10173
I20260924 19:32:23.531904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:32:23.532090 392 external_mini_cluster.cc:1428] Setting key caca6454d334b13637a0ba08bb1ed1c1
I20260924 19:32:23.533066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33407
--builtin_ntp_servers=127.0.98.20:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 19:32:23.533161 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.530839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38901 (local address 127.0.98.62:33407)
0924 19:32:23.530909 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:23.530912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:23.531136 (+ 224us) server_negotiation.cc:407] Connection header received
0924 19:32:23.531606 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:23.531608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:23.531613 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:23.531661 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:23.532920 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:23.532921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:23.532921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:32:23.532925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:23.532939 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:32:23.532942 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:23.532986 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:23.533085 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:32:23.533096 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:32:23.533341 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.530807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33407 (local address 127.0.98.1:38901)
0924 19:32:23.531019 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:23.531056 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:23.531520 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:23.531760 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:23.531765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:23.531777 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:23.532765 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:32:23.532766 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:23.532806 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:32:23.532844 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:23.532990 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:23.532994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:23.532997 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:23.533067 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:32:23.533133 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260924 19:32:23.533779 10301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33407
I20260924 19:32:23.533872 10301 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:23.534032 10301 heartbeater.cc:511] Master 127.0.98.62:33407 requested a full tablet report, sending...
I20260924 19:32:23.534587 10113 ts_manager.cc:194] Registered new tserver with Master: bb699238f6dc4648b60e37c2e6353efe (127.0.98.1:42221)
W20260924 19:32:23.612509 10304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:23.612761 10304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:23.612794 10304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:23.614071 10304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:23.614110 10304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:23.614145 10304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:32:23.615486 10304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:23.615727 10304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:23.615998 10304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:23.620913 10310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.620884 10312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.620929 10309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.621073 10304 server_base.cc:1034] running on GCE node
I20260924 19:32:23.621354 10304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:23.621570 10304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:23.622696 10304 hybrid_clock.cc:648] HybridClock initialized: now 1790278343622685 us; error 30 us; skew 500 ppm
I20260924 19:32:23.623847 10304 webserver.cc:460] Webserver started at http://127.0.98.2:41853/ using document root <none> and password file <none>
I20260924 19:32:23.624075 10304 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:23.624122 10304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:23.624258 10304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:23.625325 10304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "9b67d47599494ad18f11542e417d29e0"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "2263d61741a29890b6eb6c25e026f7ac"
server_key_iv: "75c82938ffa066c9ed69163c87d436ea"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.625629 10304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9b67d47599494ad18f11542e417d29e0"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "2263d61741a29890b6eb6c25e026f7ac"
server_key_iv: "75c82938ffa066c9ed69163c87d436ea"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.626798 10304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:32:23.627467 10318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.627641 10304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:23.627714 10304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "9b67d47599494ad18f11542e417d29e0"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "2263d61741a29890b6eb6c25e026f7ac"
server_key_iv: "75c82938ffa066c9ed69163c87d436ea"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.627767 10304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:23.631031 10304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:23.631274 10304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:23.631369 10304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:23.631523 10304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:23.631773 10304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:23.631800 10304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.631819 10304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:23.631829 10304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.637123 10304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44561
I20260924 19:32:23.637164 10431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44561 every 8 connection(s)
I20260924 19:32:23.637501 10304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:32:23.640364 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.638196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47457 (local address 127.0.98.62:33407)
0924 19:32:23.638248 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:23.638250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:23.638458 (+ 208us) server_negotiation.cc:407] Connection header received
0924 19:32:23.638978 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:23.638979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:23.638986 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:23.639037 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:23.640126 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:23.640127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:23.640127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:32:23.640132 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:23.640155 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:32:23.640159 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:23.640206 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:23.640278 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 19:32:23.640293 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:32:23.640542 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.638187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33407 (local address 127.0.98.2:47457)
0924 19:32:23.638348 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:23.638382 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:23.638878 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:23.639109 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:23.639113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:23.639121 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:23.639978 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:32:23.639978 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:23.640008 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:32:23.640037 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:23.640211 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:23.640216 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:23.640219 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:23.640285 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:32:23.640348 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260924 19:32:23.641012 10432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33407
I20260924 19:32:23.641095 10432 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:23.641232 10432 heartbeater.cc:511] Master 127.0.98.62:33407 requested a full tablet report, sending...
I20260924 19:32:23.641547 10113 ts_manager.cc:194] Registered new tserver with Master: 9b67d47599494ad18f11542e417d29e0 (127.0.98.2:44561)
I20260924 19:32:23.647686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 10304
I20260924 19:32:23.647812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:32:23.647962 392 external_mini_cluster.cc:1428] Setting key 0849fc3d6b88b2ba9cc1460fca0cdd86
I20260924 19:32:23.648968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33407
--builtin_ntp_servers=127.0.98.20:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:32:23.726253 10435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:23.726464 10435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:23.726483 10435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:23.727641 10435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:23.727680 10435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:23.727705 10435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:32:23.729065 10435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:23.729311 10435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:23.729601 10435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:23.734453 10441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:23.734524 10443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.734623 10435 server_base.cc:1034] running on GCE node
W20260924 19:32:23.734721 10440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:23.734953 10435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:23.735206 10435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:23.736374 10435 hybrid_clock.cc:648] HybridClock initialized: now 1790278343736360 us; error 27 us; skew 500 ppm
I20260924 19:32:23.737686 10435 webserver.cc:460] Webserver started at http://127.0.98.3:46831/ using document root <none> and password file <none>
I20260924 19:32:23.737905 10435 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:23.737946 10435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:23.738114 10435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:23.739187 10435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "003bd614d7204ea79653977e8b6d385d"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "e34111784d1359a5efa1b6de6dcf7255"
server_key_iv: "53c7c703126062b6130f38116ecb6508"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.739514 10435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "003bd614d7204ea79653977e8b6d385d"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "e34111784d1359a5efa1b6de6dcf7255"
server_key_iv: "53c7c703126062b6130f38116ecb6508"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.740835 10435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:23.741540 10449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.741680 10435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:23.741760 10435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "003bd614d7204ea79653977e8b6d385d"
format_stamp: "Formatted at 2026-09-24 19:32:23 on dist-test-slave-m8jb"
server_key: "e34111784d1359a5efa1b6de6dcf7255"
server_key_iv: "53c7c703126062b6130f38116ecb6508"
server_key_version: "encryptionkey@0"
I20260924 19:32:23.741827 10435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:23.745143 10435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:23.745366 10435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:23.745474 10435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:23.745647 10435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:23.745920 10435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:23.745951 10435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.745976 10435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:23.746001 10435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:23.751196 10435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42231
I20260924 19:32:23.751235 10562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42231 every 8 connection(s)
I20260924 19:32:23.751520 10435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:32:23.753098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 10435
I20260924 19:32:23.753201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:32:23.753376 392 external_mini_cluster.cc:1428] Setting key c96b3b526739738fc58b9cf447e5587f
I20260924 19:32:23.754472 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.752181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53923 (local address 127.0.98.62:33407)
0924 19:32:23.752251 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:23.752253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:23.752426 (+ 173us) server_negotiation.cc:407] Connection header received
0924 19:32:23.752923 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:23.752924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:23.752928 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:23.752982 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:23.754245 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:23.754246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:23.754247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:32:23.754251 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:23.754268 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:32:23.754272 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:23.754324 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:23.754410 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 19:32:23.754421 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:32:23.754647 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:23.752157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33407 (local address 127.0.98.3:53923)
0924 19:32:23.752317 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:23.752356 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:23.752832 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:23.753059 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:23.753063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:23.753074 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:23.754087 (+ 1013us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:32:23.754087 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:23.754126 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:32:23.754160 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:23.754332 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:23.754337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:23.754340 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:23.754409 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 19:32:23.754458 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":71,"threads_started":1}
I20260924 19:32:23.755012 10563 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33407
I20260924 19:32:23.755092 10563 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:23.755229 10563 heartbeater.cc:511] Master 127.0.98.62:33407 requested a full tablet report, sending...
I20260924 19:32:23.755541 10113 ts_manager.cc:194] Registered new tserver with Master: 003bd614d7204ea79653977e8b6d385d (127.0.98.3:42231)
I20260924 19:32:23.765170 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:32:23.766954 10152 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:32:23.766436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.0.98.62:33407)
0924 19:32:23.766478 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:23.766480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:23.766649 (+ 169us) server_negotiation.cc:407] Connection header received
0924 19:32:23.766685 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:23.766685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:23.766691 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:23.766745 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:23.766883 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48874: BlockingRecv error: recv got EOF from 127.0.0.1:48874 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:32:23.767078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 10173
I20260924 19:32:23.770855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 10304
I20260924 19:32:23.774514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 10435
I20260924 19:32:23.778004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 10082
2026-09-24T19:32:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (530 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:32:23 dist-test-slave-m8jb krb5kdc[10573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:32:23 dist-test-slave-m8jb krb5kdc[10573](info): set up 2 sockets
Sep 24 19:32:23 dist-test-slave-m8jb krb5kdc[10573](info): commencing operation
krb5kdc: starting...
W20260924 19:32:24.729156 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:32:24 dist-test-slave-m8jb krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:32:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:32:24.843899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37667
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:24.920905 10586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:24.921105 10586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:24.921133 10586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:24.922250 10586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:32:24.922290 10586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:24.922308 10586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:24.922324 10586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:32:24.922338 10586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:32:24.923753 10586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:24.923981 10586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:24.924173 10586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:24.928610 10592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:24.928608 10591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:24.928825 10586 server_base.cc:1034] running on GCE node
W20260924 19:32:24.928728 10594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:24.929118 10586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:24.929368 10586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:24.930502 10586 hybrid_clock.cc:648] HybridClock initialized: now 1790278344930490 us; error 27 us; skew 500 ppm
Sep 24 19:32:24 dist-test-slave-m8jb krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:24.932854 10586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:32:24.933262 10586 webserver.cc:460] Webserver started at http://127.0.98.62:42591/ using document root <none> and password file <none>
I20260924 19:32:24.933477 10586 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:24.933524 10586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:24.933645 10586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:24.934667 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "7934979327f344e0b70d592a7083727a"
format_stamp: "Formatted at 2026-09-24 19:32:24 on dist-test-slave-m8jb"
server_key: "1e1976456135b46231cebfce63cd23bb"
server_key_iv: "dc7907be4422b41a5501b53530706103"
server_key_version: "encryptionkey@0"
I20260924 19:32:24.934981 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "7934979327f344e0b70d592a7083727a"
format_stamp: "Formatted at 2026-09-24 19:32:24 on dist-test-slave-m8jb"
server_key: "1e1976456135b46231cebfce63cd23bb"
server_key_iv: "dc7907be4422b41a5501b53530706103"
server_key_version: "encryptionkey@0"
I20260924 19:32:24.936228 10586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:32:24.936883 10601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:24.937011 10586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:32:24.937094 10586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "7934979327f344e0b70d592a7083727a"
format_stamp: "Formatted at 2026-09-24 19:32:24 on dist-test-slave-m8jb"
server_key: "1e1976456135b46231cebfce63cd23bb"
server_key_iv: "dc7907be4422b41a5501b53530706103"
server_key_version: "encryptionkey@0"
I20260924 19:32:24.937150 10586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:24.954685 10586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:24.956477 10586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:24.956606 10586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:24.960170 10586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37667
I20260924 19:32:24.960219 10653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37667 every 8 connection(s)
I20260924 19:32:24.960511 10586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:32:24.961488 10654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:24.963665 10654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a: Bootstrap starting.
I20260924 19:32:24.964205 10654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:24.964457 10654 log.cc:824] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:24.965157 10654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a: No bootstrap required, opened a new log
I20260924 19:32:24.966679 10654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7934979327f344e0b70d592a7083727a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } }
I20260924 19:32:24.966815 10654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:24.966848 10654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7934979327f344e0b70d592a7083727a, State: Initialized, Role: FOLLOWER
I20260924 19:32:24.966995 10654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7934979327f344e0b70d592a7083727a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } }
I20260924 19:32:24.967060 10654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:32:24.967092 10654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:32:24.967137 10654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:24.967900 10654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7934979327f344e0b70d592a7083727a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } }
I20260924 19:32:24.967988 10654 leader_election.cc:302] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7934979327f344e0b70d592a7083727a; no voters:
I20260924 19:32:24.968143 10654 leader_election.cc:288] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:32:24.968180 10657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:24.968376 10654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:32:24.968382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 10586
I20260924 19:32:24.968466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:32:24.968673 392 external_mini_cluster.cc:1428] Setting key 34335c6f4b1f9e481be495e449e70991
I20260924 19:32:24.968919 10657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [term 1 LEADER]: Becoming Leader. State: Replica: 7934979327f344e0b70d592a7083727a, State: Running, Role: LEADER
I20260924 19:32:24.969043 10657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7934979327f344e0b70d592a7083727a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } }
I20260924 19:32:24.969547 10661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7934979327f344e0b70d592a7083727a. Latest consensus state: current_term: 1 leader_uuid: "7934979327f344e0b70d592a7083727a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7934979327f344e0b70d592a7083727a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } } }
I20260924 19:32:24.969637 10658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7934979327f344e0b70d592a7083727a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7934979327f344e0b70d592a7083727a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37667 } } }
I20260924 19:32:24.969725 10661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [sys.catalog]: This master's current role is: LEADER
I20260924 19:32:24.970062 10658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a [sys.catalog]: This master's current role is: LEADER
W20260924 19:32:24.970095 10673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:32:24.970129 10673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:32:24.970345 10676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:32:24.970569 10676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:32:24.971948 10676 catalog_manager.cc:1347] Generated new cluster ID: 78d6f50cd0d74676b593652c8e2109cd
I20260924 19:32:24.971997 10676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:32:24 dist-test-slave-m8jb krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:32:24.978032 10676 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:32:24.978696 10676 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:32:24.980659 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:24.969296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.0.98.62:37667)
0924 19:32:24.969941 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:24.969950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:24.969981 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:32:24.970218 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:24.970226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:24.970284 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:24.970423 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:24.971156 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:24.972501 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:24.973565 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:24.974207 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:24.974312 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:24.977027 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:24.977031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:24.977032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:24.977044 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:24.979081 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:24.979471 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:24.979474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:24.979474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:24.979584 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:24.979761 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:24.979762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:24.979762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:24.980014 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:24.980168 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:24.980269 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 19:32:24.980412 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":73,"threads_started":1}
I20260924 19:32:24.984270 10676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a: Generated new TSK 0
I20260924 19:32:24.984426 10676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:32:25.015564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:25.103647 10680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:25.103852 10680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:25.103879 10680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:25.105073 10680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:25.105115 10680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:25.105141 10680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:32:25.106547 10680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:25.106770 10680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:25.106986 10680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:25.111586 10688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:25.111671 10686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:25.111594 10685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:25.111763 10680 server_base.cc:1034] running on GCE node
I20260924 19:32:25.112004 10680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:25.112269 10680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:25.113396 10680 hybrid_clock.cc:648] HybridClock initialized: now 1790278345113382 us; error 28 us; skew 500 ppm
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:25.116752 10680 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:32:25.117262 10680 webserver.cc:460] Webserver started at http://127.0.98.1:42595/ using document root <none> and password file <none>
I20260924 19:32:25.117506 10680 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:25.117565 10680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:25.117705 10680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:25.119093 10680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "d3f2dd4f2c3749278a8189e1a8ed652e"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "1a98613b86782b9c9c0310db9e86c5a7"
server_key_iv: "adc0cc55b5abcd056279d401d20ac603"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.119484 10680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d3f2dd4f2c3749278a8189e1a8ed652e"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "1a98613b86782b9c9c0310db9e86c5a7"
server_key_iv: "adc0cc55b5abcd056279d401d20ac603"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.121028 10680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:25.121770 10695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.121896 10680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:32:25.121974 10680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "d3f2dd4f2c3749278a8189e1a8ed652e"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "1a98613b86782b9c9c0310db9e86c5a7"
server_key_iv: "adc0cc55b5abcd056279d401d20ac603"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.122040 10680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:25.138604 10680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:25.140751 10680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:25.140913 10680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:25.141122 10680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:25.141407 10680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:25.141438 10680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.141464 10680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:25.141480 10680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.146752 10680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39255
I20260924 19:32:25.146790 10808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39255 every 8 connection(s)
I20260924 19:32:25.147081 10680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:32:25.150477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 10680
I20260924 19:32:25.150556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:32:25.150718 392 external_mini_cluster.cc:1428] Setting key 30b24b11ac5201b6b6293af1b4acef8d
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:32:25.156432 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.147694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59705 (local address 127.0.98.62:37667)
0924 19:32:25.147767 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.147769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.147914 (+ 145us) server_negotiation.cc:407] Connection header received
0924 19:32:25.148385 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.148386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.148395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.148433 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:25.149372 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.149875 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.150949 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.151237 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.151307 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.153769 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:25.153771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:25.153771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:25.153779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:25.155149 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:25.155506 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:25.155507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:25.155507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:25.155576 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:25.155798 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:25.155799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:25.155799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:25.155942 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:25.156005 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.156227 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.156336 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:32:25.156461 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.147658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.1:59705)
0924 19:32:25.147808 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:25.147843 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:25.148302 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:25.148487 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:25.148490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:25.149091 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:25.149289 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.149293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.149948 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.149949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.150874 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.150877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.151313 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.151314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.151445 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.151902 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:25.151912 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:25.153679 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:25.155223 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:25.155228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:25.155230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:25.155435 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:25.155634 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:25.155634 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:25.155635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:25.155741 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:25.156002 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:25.156004 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:25.156114 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 19:32:25.156189 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:32:25.156304 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260924 19:32:25.156961 10809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37667
I20260924 19:32:25.157047 10809 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:25.157210 10809 heartbeater.cc:511] Master 127.0.98.62:37667 requested a full tablet report, sending...
I20260924 19:32:25.157733 10618 ts_manager.cc:194] Registered new tserver with Master: d3f2dd4f2c3749278a8189e1a8ed652e (127.0.98.1:39255)
I20260924 19:32:25.158619 10618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59705
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:32:25.176205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:25.256713 10814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:25.256927 10814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:25.256949 10814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:25.258168 10814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:25.258224 10814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:25.258251 10814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:32:25.259770 10814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:25.260007 10814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:25.260233 10814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:25.265089 10822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:25.265270 10814 server_base.cc:1034] running on GCE node
W20260924 19:32:25.265097 10819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:25.265101 10820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:25.265524 10814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:25.265729 10814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:25.266887 10814 hybrid_clock.cc:648] HybridClock initialized: now 1790278345266877 us; error 32 us; skew 500 ppm
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:25.269546 10814 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:32:25.270002 10814 webserver.cc:460] Webserver started at http://127.0.98.2:37213/ using document root <none> and password file <none>
I20260924 19:32:25.270222 10814 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:25.270267 10814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:25.270386 10814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:25.271446 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "2eaf4c3adf07453ba69f578d5927fa9a"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "eda6020ffdc66f634fa6fa2e7b43aaf9"
server_key_iv: "d1dd72780b3c646c7981bd8028d918f1"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.271760 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2eaf4c3adf07453ba69f578d5927fa9a"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "eda6020ffdc66f634fa6fa2e7b43aaf9"
server_key_iv: "d1dd72780b3c646c7981bd8028d918f1"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.273048 10814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:32:25.273758 10829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.273891 10814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:32:25.273968 10814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "2eaf4c3adf07453ba69f578d5927fa9a"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "eda6020ffdc66f634fa6fa2e7b43aaf9"
server_key_iv: "d1dd72780b3c646c7981bd8028d918f1"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.274031 10814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:25.296908 10814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:25.298998 10814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:25.299110 10814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:25.299337 10814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:25.299618 10814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:25.299649 10814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.299678 10814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:25.299695 10814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.305222 10814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41493
I20260924 19:32:25.305275 10942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41493 every 8 connection(s)
I20260924 19:32:25.305578 10814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:32:25.311831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 10814
I20260924 19:32:25.311919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:32:25.312117 392 external_mini_cluster.cc:1428] Setting key c78c2825d7ec4549658cd004516980d3
I20260924 19:32:25.314721 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.306309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53779 (local address 127.0.98.62:37667)
0924 19:32:25.306387 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.306389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.306545 (+ 156us) server_negotiation.cc:407] Connection header received
0924 19:32:25.307032 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.307033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.307044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.307113 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:25.308028 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.308559 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.309593 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.309803 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.309864 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.311808 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:25.311810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:25.311810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:25.311816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:25.313260 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:25.313720 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:25.313721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:25.313721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:25.313779 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:25.314064 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:25.314066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:25.314066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:25.314219 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:25.314297 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.314528 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.314631 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 19:32:25.314777 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.306280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.2:53779)
0924 19:32:25.306434 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:25.306469 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:25.306926 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:25.307197 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:25.307200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:25.307736 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:25.307925 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.307929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.308659 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.308661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.309499 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.309501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.309872 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.309872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.309976 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.310431 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:25.310439 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:25.311726 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:25.313342 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:25.313348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:25.313351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:25.313618 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:25.313842 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:25.313842 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:25.313843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:25.313966 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:25.314291 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:25.314295 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:25.314412 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 19:32:25.314495 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:32:25.314610 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260924 19:32:25.315176 10943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37667
I20260924 19:32:25.315270 10943 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:25.315446 10943 heartbeater.cc:511] Master 127.0.98.62:37667 requested a full tablet report, sending...
I20260924 19:32:25.315805 10618 ts_manager.cc:194] Registered new tserver with Master: 2eaf4c3adf07453ba69f578d5927fa9a (127.0.98.2:41493)
I20260924 19:32:25.316205 10618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53779
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:32:25.336913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:32:25.416666 10948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:32:25.416886 10948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:32:25.416909 10948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:32:25.418056 10948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:32:25.418102 10948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:32:25.418129 10948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:32:25.419543 10948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:32:25.419760 10948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:32:25.419956 10948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:32:25.424763 10956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:25.424811 10954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:32:25.424801 10953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:32:25.424855 10948 server_base.cc:1034] running on GCE node
I20260924 19:32:25.425145 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:32:25.425382 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:32:25.426522 10948 hybrid_clock.cc:648] HybridClock initialized: now 1790278345426505 us; error 27 us; skew 500 ppm
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:32:25.428920 10948 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:32:25.429352 10948 webserver.cc:460] Webserver started at http://127.0.98.3:45761/ using document root <none> and password file <none>
I20260924 19:32:25.429530 10948 fs_manager.cc:359] Metadata directory not provided
I20260924 19:32:25.429569 10948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:32:25.429661 10948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:32:25.430657 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "81389ba6a5274e95bae7cab02cdfaea7"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "e36bf6dd944033c86dd97d88ba6f07fc"
server_key_iv: "461c97f61a7dcdf213445c3e27665bdd"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.430974 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "81389ba6a5274e95bae7cab02cdfaea7"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "e36bf6dd944033c86dd97d88ba6f07fc"
server_key_iv: "461c97f61a7dcdf213445c3e27665bdd"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.432224 10948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:32:25.432987 10963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.433187 10948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:32:25.433286 10948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "81389ba6a5274e95bae7cab02cdfaea7"
format_stamp: "Formatted at 2026-09-24 19:32:25 on dist-test-slave-m8jb"
server_key: "e36bf6dd944033c86dd97d88ba6f07fc"
server_key_iv: "461c97f61a7dcdf213445c3e27665bdd"
server_key_version: "encryptionkey@0"
I20260924 19:32:25.433350 10948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:32:25.439997 10948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:32:25.442050 10948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:32:25.442152 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:32:25.442319 10948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:32:25.442601 10948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:32:25.442632 10948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.442651 10948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:32:25.442672 10948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:32:25.447947 10948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35817
I20260924 19:32:25.448006 11076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35817 every 8 connection(s)
I20260924 19:32:25.448292 10948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:32:25.451434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 10948
I20260924 19:32:25.451532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:32:25.451696 392 external_mini_cluster.cc:1428] Setting key c941dcf7be6a19e247f357a290452dd6
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:32:25.456794 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.448977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47359 (local address 127.0.98.62:37667)
0924 19:32:25.449033 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.449035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.449229 (+ 194us) server_negotiation.cc:407] Connection header received
0924 19:32:25.449693 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.449694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.449704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.449765 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:25.450667 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.451171 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.452225 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.452440 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.452495 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.454516 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:25.454517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:25.454518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:25.454524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:25.455574 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:25.455938 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:25.455939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:25.455939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:25.455992 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:25.456236 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:25.456237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:25.456237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:25.456336 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:25.456380 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.456619 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.456730 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:32:25.456943 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.448958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37667 (local address 127.0.98.3:47359)
0924 19:32:25.449109 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:25.449151 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:25.449615 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:25.449820 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:25.449824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:25.450392 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:32:25.450581 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.450586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.451270 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.451271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.452165 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.452167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.452513 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.452515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.452662 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.453135 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:32:25.453145 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:32:25.454437 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:32:25.455654 (+ 1217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:25.455659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:25.455661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:25.455877 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:25.456069 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:32:25.456070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:32:25.456070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:32:25.456181 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:32:25.456381 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:32:25.456384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:32:25.456497 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 19:32:25.456574 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:32:25.456754 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260924 19:32:25.457392 11077 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37667
I20260924 19:32:25.457482 11077 heartbeater.cc:464] Registering TS with master...
I20260924 19:32:25.457655 11077 heartbeater.cc:511] Master 127.0.98.62:37667 requested a full tablet report, sending...
I20260924 19:32:25.457974 10618 ts_manager.cc:194] Registered new tserver with Master: 81389ba6a5274e95bae7cab02cdfaea7 (127.0.98.3:35817)
I20260924 19:32:25.458320 10618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47359
I20260924 19:32:25.463284 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:32:25.470379 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.464516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.0.98.62:37667)
0924 19:32:25.464580 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.464582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.464687 (+ 105us) server_negotiation.cc:407] Connection header received
0924 19:32:25.464756 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.464757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.464766 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.464811 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:32:25.465330 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.465812 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.466527 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.466716 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.466768 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.467886 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:32:25.467888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:32:25.467888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:32:25.467895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:32:25.469237 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:25.469574 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:25.469576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:25.469576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:25.469627 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:32:25.469799 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:32:25.469800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:32:25.469800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:32:25.469948 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:32:25.470041 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.470202 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.470293 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:32:25.471844 10618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:32:25.473491 10618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:32:25.479645 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.475637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39255 (local address 127.0.0.1:36568)
0924 19:32:25.475875 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:25.475885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:25.475937 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:25.476274 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:25.476277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:25.476279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:25.476344 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.476346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.477375 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.477376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.478560 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.478562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.479352 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.479353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.479438 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.479453 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:32:25.479522 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 19:32:25.479568 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":52,"threads_started":1}
I20260924 19:32:25.479671 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.475896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.0.98.1:39255)
0924 19:32:25.476046 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.476050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.476060 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:32:25.476098 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.476101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.476117 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.476248 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:25.476424 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.477249 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.478630 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.479268 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.479349 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.479409 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.479552 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.479592 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260924 19:32:25.480134 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.475637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41493 (local address 127.0.0.1:41236)
0924 19:32:25.476023 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:25.476031 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:25.476069 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:25.476531 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:25.476532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:25.476534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:25.476569 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.476571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.477886 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.477887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.478853 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.478855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.479866 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.479867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.479932 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.479936 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:32:25.479966 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 19:32:25.480062 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260924 19:32:25.480484 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.476080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.0.98.2:41493)
0924 19:32:25.476292 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.476296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.476369 (+ 73us) server_negotiation.cc:407] Connection header received
0924 19:32:25.476417 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.476420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.476435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.476671 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:25.476748 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.477782 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.478928 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.479633 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.480265 (+ 632us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.480335 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.480363 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.480404 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":82,"threads_started":1}
I20260924 19:32:25.480834 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.475629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35817 (local address 127.0.0.1:37458)
0924 19:32:25.475752 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:25.475787 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:25.475844 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:25.476274 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:25.476279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:25.476282 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:25.476356 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.476358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.478917 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.478918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.479689 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.479692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.480648 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.480650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.480714 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.480718 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:32:25.480748 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 19:32:25.480786 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260924 19:32:25.480726 10743 tablet_service.cc:1467] Processing CreateTablet for tablet 2759b78b919443d6a455076f5dbe6721 (DEFAULT_TABLE table=test-table [id=9adb14c8bf3c458dab27b4e70da6149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:25.481104 10743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2759b78b919443d6a455076f5dbe6721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:25.481119 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.475897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37458 (local address 127.0.98.3:35817)
0924 19:32:25.476100 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.476105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.476116 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:32:25.476155 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.476157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.476172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.476443 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:25.476573 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.477752 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.479772 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.480539 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.480938 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.481001 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.481025 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.481062 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":89,"threads_started":1}
I20260924 19:32:25.481700 11011 tablet_service.cc:1467] Processing CreateTablet for tablet 2759b78b919443d6a455076f5dbe6721 (DEFAULT_TABLE table=test-table [id=9adb14c8bf3c458dab27b4e70da6149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:25.481964 11011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2759b78b919443d6a455076f5dbe6721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:25.483789 11091 tablet_bootstrap.cc:492] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: Bootstrap starting.
I20260924 19:32:25.484526 11091 tablet_bootstrap.cc:654] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:25.484579 10877 tablet_service.cc:1467] Processing CreateTablet for tablet 2759b78b919443d6a455076f5dbe6721 (DEFAULT_TABLE table=test-table [id=9adb14c8bf3c458dab27b4e70da6149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:32:25.484930 10877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2759b78b919443d6a455076f5dbe6721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:32:25.485013 11092 tablet_bootstrap.cc:492] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: Bootstrap starting.
I20260924 19:32:25.485246 11091 log.cc:824] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:25.485575 11092 tablet_bootstrap.cc:654] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:25.486025 11092 log.cc:824] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:25.486048 11091 tablet_bootstrap.cc:492] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: No bootstrap required, opened a new log
I20260924 19:32:25.486112 11091 ts_tablet_manager.cc:1397] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:32:25.488111 11095 tablet_bootstrap.cc:492] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Bootstrap starting.
I20260924 19:32:25.488137 11091 raft_consensus.cc:348] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.488327 11091 raft_consensus.cc:374] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:25.488374 11091 raft_consensus.cc:729] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f2dd4f2c3749278a8189e1a8ed652e, State: Initialized, Role: FOLLOWER
I20260924 19:32:25.488492 11091 consensus_queue.cc:260] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.488693 11095 tablet_bootstrap.cc:654] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Neither blocks nor log segments found. Creating new log.
I20260924 19:32:25.488793 11091 ts_tablet_manager.cc:1428] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 19:32:25.488798 10809 heartbeater.cc:503] Master 127.0.98.62:37667 was elected leader, sending a full tablet report...
I20260924 19:32:25.489223 11095 log.cc:824] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:32:25.489460 11092 tablet_bootstrap.cc:492] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: No bootstrap required, opened a new log
I20260924 19:32:25.489527 11092 ts_tablet_manager.cc:1397] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 19:32:25.489986 11095 tablet_bootstrap.cc:492] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: No bootstrap required, opened a new log
I20260924 19:32:25.490051 11095 ts_tablet_manager.cc:1397] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:32:25.491441 11092 raft_consensus.cc:348] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.491609 11092 raft_consensus.cc:374] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:25.491642 11092 raft_consensus.cc:729] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81389ba6a5274e95bae7cab02cdfaea7, State: Initialized, Role: FOLLOWER
I20260924 19:32:25.491765 11092 consensus_queue.cc:260] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.491832 11095 raft_consensus.cc:348] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.491999 11095 raft_consensus.cc:374] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:32:25.492035 11092 ts_tablet_manager.cc:1428] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:32:25.492039 11095 raft_consensus.cc:729] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eaf4c3adf07453ba69f578d5927fa9a, State: Initialized, Role: FOLLOWER
I20260924 19:32:25.492054 11077 heartbeater.cc:503] Master 127.0.98.62:37667 was elected leader, sending a full tablet report...
I20260924 19:32:25.492136 11095 consensus_queue.cc:260] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.492380 11095 ts_tablet_manager.cc:1428] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:32:25.492403 10943 heartbeater.cc:503] Master 127.0.98.62:37667 was elected leader, sending a full tablet report...
W20260924 19:32:25.556118 10944 tablet.cc:2369] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:32:25.567476 11099 raft_consensus.cc:482] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:32:25.567605 11099 raft_consensus.cc:504] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.567950 11099 leader_election.cc:288] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81389ba6a5274e95bae7cab02cdfaea7 (127.0.98.3:35817), d3f2dd4f2c3749278a8189e1a8ed652e (127.0.98.1:39255)
I20260924 19:32:25.570930 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.568217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57783 (local address 127.0.98.3:35817)
0924 19:32:25.568284 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.568286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.568294 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:32:25.568325 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.568325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.568336 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.568415 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:25.568585 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.569258 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.570235 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.570609 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.570674 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.570684 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.570820 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.570855 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:32:25.570935 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.568223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42167 (local address 127.0.98.1:39255)
0924 19:32:25.568264 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.568266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.568437 (+ 171us) server_negotiation.cc:407] Connection header received
0924 19:32:25.568467 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.568468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.568484 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.568542 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:32:25.568767 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.569228 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.570230 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.570594 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.570648 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.570658 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.570819 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.570859 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:32:25.570945 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.568144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35817 (local address 127.0.98.2:57783)
0924 19:32:25.568200 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:25.568212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:25.568267 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:25.568423 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:25.568424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:25.568426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:25.568513 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.568515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.569345 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.569346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.570148 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.570150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.570685 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.570686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.570753 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.570756 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:25.570802 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:32:25.570842 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":17}
I20260924 19:32:25.570945 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.568144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39255 (local address 127.0.98.2:42167)
0924 19:32:25.568358 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 19:32:25.568368 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:32:25.568423 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:32:25.568549 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:32:25.568550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:32:25.568551 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:32:25.568685 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.568687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.569286 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.569287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.570150 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:32:25.570152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.570677 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.570677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:32:25.570749 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.570752 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:32:25.570797 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:32:25.570838 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":42,"thread_start_us":118,"threads_started":1}
I20260924 19:32:25.571310 11031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2759b78b919443d6a455076f5dbe6721" candidate_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81389ba6a5274e95bae7cab02cdfaea7" is_pre_election: true
I20260924 19:32:25.571483 11031 raft_consensus.cc:2454] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eaf4c3adf07453ba69f578d5927fa9a in term 0.
I20260924 19:32:25.571576 10763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2759b78b919443d6a455076f5dbe6721" candidate_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" is_pre_election: true
I20260924 19:32:25.571714 10763 raft_consensus.cc:2454] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2eaf4c3adf07453ba69f578d5927fa9a in term 0.
I20260924 19:32:25.571732 10832 leader_election.cc:302] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eaf4c3adf07453ba69f578d5927fa9a, 81389ba6a5274e95bae7cab02cdfaea7; no voters:
I20260924 19:32:25.571851 11099 raft_consensus.cc:2781] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:32:25.571921 11099 raft_consensus.cc:482] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:32:25.571943 11099 raft_consensus.cc:3037] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:25.572623 11099 raft_consensus.cc:504] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.572762 11099 leader_election.cc:288] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [CANDIDATE]: Term 1 election: Requested vote from peers 81389ba6a5274e95bae7cab02cdfaea7 (127.0.98.3:35817), d3f2dd4f2c3749278a8189e1a8ed652e (127.0.98.1:39255)
I20260924 19:32:25.572896 11031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2759b78b919443d6a455076f5dbe6721" candidate_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81389ba6a5274e95bae7cab02cdfaea7"
I20260924 19:32:25.572921 10763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2759b78b919443d6a455076f5dbe6721" candidate_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e"
I20260924 19:32:25.572974 11031 raft_consensus.cc:3037] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:25.572984 10763 raft_consensus.cc:3037] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:32:25.573616 10763 raft_consensus.cc:2454] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eaf4c3adf07453ba69f578d5927fa9a in term 1.
I20260924 19:32:25.573662 11031 raft_consensus.cc:2454] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2eaf4c3adf07453ba69f578d5927fa9a in term 1.
I20260924 19:32:25.573800 10833 leader_election.cc:302] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eaf4c3adf07453ba69f578d5927fa9a, d3f2dd4f2c3749278a8189e1a8ed652e; no voters:
I20260924 19:32:25.573895 11099 raft_consensus.cc:2781] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:32:25.574043 11099 raft_consensus.cc:686] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 1 LEADER]: Becoming Leader. State: Replica: 2eaf4c3adf07453ba69f578d5927fa9a, State: Running, Role: LEADER
I20260924 19:32:25.574262 11099 consensus_queue.cc:237] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } }
I20260924 19:32:25.575045 10618 catalog_manager.cc:5495] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2eaf4c3adf07453ba69f578d5927fa9a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2eaf4c3adf07453ba69f578d5927fa9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 } health_report { overall_health: UNKNOWN } } }
I20260924 19:32:25.601684 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:32:25.599380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.0.98.2:41493)
0924 19:32:25.599434 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:32:25.599436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:32:25.599444 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:32:25.599478 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:32:25.599479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:32:25.599500 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:32:25.599566 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:32:25.599681 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.600176 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.600914 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:32:25.601114 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:32:25.601171 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:32:25.601342 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:32:25.601458 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:32:25.601531 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:32:25.601569 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 19:32:25.601587 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:32:25.603367 11031 raft_consensus.cc:1261] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 1 FOLLOWER]: Refusing update from remote peer 2eaf4c3adf07453ba69f578d5927fa9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:25.603367 10763 raft_consensus.cc:1261] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 1 FOLLOWER]: Refusing update from remote peer 2eaf4c3adf07453ba69f578d5927fa9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:32:25.603700 11101 consensus_queue.cc:1035] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81389ba6a5274e95bae7cab02cdfaea7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:25.603829 11099 consensus_queue.cc:1035] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f2dd4f2c3749278a8189e1a8ed652e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:32:25.606101 11104 mvcc.cc:205] Tried to move back new op lower bound from 7332980103588945920 to 7332980103472640000. Current Snapshot: MvccSnapshot[applied={T|T < 7332980103588945920}]
I20260924 19:32:25.606177 11105 mvcc.cc:205] Tried to move back new op lower bound from 7332980103588945920 to 7332980103472640000. Current Snapshot: MvccSnapshot[applied={T|T < 7332980103588945920}]
I20260924 19:32:25.607672 11107 mvcc.cc:205] Tried to move back new op lower bound from 7332980103588945920 to 7332980103472640000. Current Snapshot: MvccSnapshot[applied={T|T < 7332980103588945920 or (T in {7332980103588945920})}]
I20260924 19:32:25.610319 10617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:25.610415 10617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:32:25.611743 10617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7934979327f344e0b70d592a7083727a: Sending DeleteTablet for 3 replicas of tablet 2759b78b919443d6a455076f5dbe6721
I20260924 19:32:25.612113 10877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2759b78b919443d6a455076f5dbe6721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41236
I20260924 19:32:25.612149 10743 tablet_service.cc:1514] Processing DeleteTablet for tablet 2759b78b919443d6a455076f5dbe6721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36568
I20260924 19:32:25.612353 11113 tablet_replica.cc:330] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: stopping tablet replica
I20260924 19:32:25.612452 11113 raft_consensus.cc:2229] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:25.612560 11113 raft_consensus.cc:2258] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:25.612551 11114 tablet_replica.cc:330] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: stopping tablet replica
I20260924 19:32:25.612712 11114 raft_consensus.cc:2229] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:32:25.612846 11114 raft_consensus.cc:2258] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:25.613281 11114 ts_tablet_manager.cc:1905] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:25.613349 11011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2759b78b919443d6a455076f5dbe6721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:32:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37458
I20260924 19:32:25.613919 11113 ts_tablet_manager.cc:1905] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:25.615113 11114 ts_tablet_manager.cc:1918] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:25.615239 11114 log.cc:1196] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/2759b78b919443d6a455076f5dbe6721
I20260924 19:32:25.615415 11113 ts_tablet_manager.cc:1918] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:25.615485 11113 log.cc:1196] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/2759b78b919443d6a455076f5dbe6721
I20260924 19:32:25.615664 11114 ts_tablet_manager.cc:1939] T 2759b78b919443d6a455076f5dbe6721 P 2eaf4c3adf07453ba69f578d5927fa9a: Deleting consensus metadata
I20260924 19:32:25.615763 11113 ts_tablet_manager.cc:1939] T 2759b78b919443d6a455076f5dbe6721 P d3f2dd4f2c3749278a8189e1a8ed652e: Deleting consensus metadata
I20260924 19:32:25.616083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 10680
I20260924 19:32:25.616117 10602 catalog_manager.cc:4841] TS 2eaf4c3adf07453ba69f578d5927fa9a (127.0.98.2:41493): tablet 2759b78b919443d6a455076f5dbe6721 (table test-table [id=9adb14c8bf3c458dab27b4e70da6149b]) successfully deleted
I20260924 19:32:25.616164 10605 catalog_manager.cc:4841] TS d3f2dd4f2c3749278a8189e1a8ed652e (127.0.98.1:39255): tablet 2759b78b919443d6a455076f5dbe6721 (table test-table [id=9adb14c8bf3c458dab27b4e70da6149b]) successfully deleted
I20260924 19:32:25.617375 11115 tablet_replica.cc:330] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: stopping tablet replica
I20260924 19:32:25.617457 11115 raft_consensus.cc:2229] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:32:25.617529 11115 raft_consensus.cc:2258] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:32:25.621439 11115 ts_tablet_manager.cc:1905] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:32:25.621937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 10814
I20260924 19:32:25.622816 11115 ts_tablet_manager.cc:1918] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:32:25.622880 11115 log.cc:1196] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/2759b78b919443d6a455076f5dbe6721
I20260924 19:32:25.623098 11115 ts_tablet_manager.cc:1939] T 2759b78b919443d6a455076f5dbe6721 P 81389ba6a5274e95bae7cab02cdfaea7: Deleting consensus metadata
I20260924 19:32:25.623477 10604 catalog_manager.cc:4841] TS 81389ba6a5274e95bae7cab02cdfaea7 (127.0.98.3:35817): tablet 2759b78b919443d6a455076f5dbe6721 (table test-table [id=9adb14c8bf3c458dab27b4e70da6149b]) successfully deleted
I20260924 19:32:25.626423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 10948
I20260924 19:32:25.630189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 10586
2026-09-24T19:32:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1852 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[11118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[11118](info): set up 2 sockets
Sep 24 19:32:25 dist-test-slave-m8jb krb5kdc[11118](info): commencing operation
krb5kdc: starting...
W20260924 19:32:26.577204 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:32:26 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:32:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 19:32:26.732221 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLXPSqI/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/postgres -l logfile start
2026-09-24 19:32:29.157 UTC [11148] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:32:29.157 UTC [11148] LOG: listening on IPv4 address "127.0.98.20", port 52651
2026-09-24 19:32:29.161 UTC [11148] LOG: listening on Unix socket "/tmp/.s.PGSQL.52651"
2026-09-24 19:32:29.181 UTC [11151] LOG: database system was shut down at 2026-09-24 19:32:27 UTC
2026-09-24 19:32:29.185 UTC [11148] LOG: database system is ready to accept connections
I20260924 19:32:30.053156 392 mini_postgres.cc:96] Postgres bound to 52651
2026-09-24 19:32:30.058 UTC [11159] FATAL: database "slave" does not exist
127.0.98.20:52651 - accepting connections
I20260924 19:32:30.059290 392 mini_ranger.cc:161] Starting Ranger...
I20260924 19:32:30.073113 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:32:30.559792 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:32:30.559877 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-admin
2026-09-24 19:32:30,858 [I] DB FLAVOR :POSTGRES
2026-09-24 19:32:30,858 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:32:30,858 [I] Checking connection..
2026-09-24 19:32:30,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:32:31,104 [I] Checking connection passed.
2026-09-24 19:32:31,104 [I] --------- Verifying version history table ---------
2026-09-24 19:32:31,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:32:31,360 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:32:31,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:32:31,615 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:32:31,615 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:32:31,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:32:31.836 UTC [11223] WARNING: there is no transaction in progress
2026-09-24 19:32:31,841 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:32:31,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:32:32,097 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:32:32,097 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:32:32,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:32:32,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:32:32,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m8jb.c.gcp-upstream.internal','N') ;"
2026-09-24 19:32:32,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:32:33,039 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:32:33,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:32:33,292 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:32:33,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:32:33,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:32:33,757 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:32:33,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:32:34.718 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.735 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.751 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.768 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.785 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.802 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.856 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.863 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.878 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.889 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.899 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.914 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.924 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.931 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.938 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:34.945 UTC [11331] WARNING: there is no transaction in progress
2026-09-24 19:32:35,581 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:32:35,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:32:35,816 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:32:35,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:32:36,038 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:32:36,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:32:36,275 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:32:36,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:32:36,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:32:36,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m8jb.c.gcp-upstream.internal';"
2026-09-24 19:32:37,023 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:32:37,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:32:37,260 [I] DB_PATCHES have already been applied
I20260924 19:32:37.266249 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLXPSqI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:32:37.266324 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 19:32:37.269311 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54203
Sep 24, 2026 7:32:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:32:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:32:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:32:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:32:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:32:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:32:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:32:38.278934 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.220s user 0.000s sys 0.008s
Sep 24, 2026 7:32:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:32:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:32:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:32:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:32:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:32:58.713924 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:32:58 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278378, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:32:58.791065 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:32:58.799592 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:32:59.670751 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:32:59.670820 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms
I20260924 19:32:59.670836 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52651
2026-09-24 19:32:59,829 [I] DB FLAVOR :POSTGRES
2026-09-24 19:32:59,830 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:32:59,830 [I] Checking connection
2026-09-24 19:32:59,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:33:00,071 [I] connection success
2026-09-24 19:33:00,072 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:33:00,072 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:33:00,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:33:00,317 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:33:00,317 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:33:00,317 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:33:00,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52651/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:33:00,597 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:33:01.063766 392 mini_ranger_kms.cc:310] Created kms service
I20260924 19:33:01.272776 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:33:01.316074 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:33:01 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278378, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:01.685061 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:33:01.685180 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:33:01.685199 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 19:33:01.688172 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11511
I20260924 19:33:01.688295 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33585
Sep 24, 2026 7:33:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:33:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:33:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:33:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:33:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:33:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:33:03.760952 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.970s user 0.000s sys 0.141s
I20260924 19:33:03.761062 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:33:03.761137 392 mini_ranger_kms.cc:401] 127.0.98.20:33585/kms/v1/keys
Sep 24 19:33:03 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:33:04 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 19:33:04 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278383, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 19:33:06 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:33:06.859535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32879
--ranger_config_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33585/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:06.938122 11562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:06.938323 11562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:06.938344 11562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:06.939491 11562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:33:06.939531 11562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:06.939544 11562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:06.939555 11562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:33:06.939570 11562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:33:06.941151 11562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33585/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32879
--ranger_config_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:06.941378 11562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:06.941591 11562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:06.947218 11568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:06.947230 11567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:06.947310 11570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:06.947602 11562 server_base.cc:1034] running on GCE node
I20260924 19:33:06.947846 11562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:06.948115 11562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:06.949246 11562 hybrid_clock.cc:648] HybridClock initialized: now 1790278386949234 us; error 31 us; skew 500 ppm
Sep 24 19:33:06 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:06.957050 11562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:33:06.957523 11562 webserver.cc:460] Webserver started at http://127.0.98.62:43953/ using document root <none> and password file <none>
I20260924 19:33:06.957736 11562 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:06.957775 11562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:06.957895 11562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:33:06 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:07.017663 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "6bf00dcc61364d4c8ca12f0a116e827a"
format_stamp: "Formatted at 2026-09-24 19:33:07 on dist-test-slave-m8jb"
server_key: "4e6e33745551932faee94eceae044cf9"
server_key_iv: "4fb442d42c10e55c04ef55166d1e0fc9"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:07.018043 11562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "6bf00dcc61364d4c8ca12f0a116e827a"
format_stamp: "Formatted at 2026-09-24 19:33:07 on dist-test-slave-m8jb"
server_key: "4e6e33745551932faee94eceae044cf9"
server_key_iv: "4fb442d42c10e55c04ef55166d1e0fc9"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:07.019421 11562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 19:33:07.034714 11577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:07.035077 11562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:33:07.035177 11562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "6bf00dcc61364d4c8ca12f0a116e827a"
format_stamp: "Formatted at 2026-09-24 19:33:07 on dist-test-slave-m8jb"
server_key: "4e6e33745551932faee94eceae044cf9"
server_key_iv: "4fb442d42c10e55c04ef55166d1e0fc9"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:07.035276 11562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:07.048557 11562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:07.050400 11562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:07.050522 11562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:07.055217 11562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32879
I20260924 19:33:07.055240 11629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32879 every 8 connection(s)
I20260924 19:33:07.055574 11562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:33:07.056182 11630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:07.058637 11630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a: Bootstrap starting.
I20260924 19:33:07.059187 11630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:07.059844 11630 log.cc:824] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:07.060468 11630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a: No bootstrap required, opened a new log
I20260924 19:33:07.062417 11630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf00dcc61364d4c8ca12f0a116e827a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32879 } }
I20260924 19:33:07.062564 11630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:07.062585 11630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf00dcc61364d4c8ca12f0a116e827a, State: Initialized, Role: FOLLOWER
I20260924 19:33:07.062711 11630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf00dcc61364d4c8ca12f0a116e827a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32879 } }
I20260924 19:33:07.062779 11630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:33:07.062803 11630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:33:07.062847 11630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:07.063354 11630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf00dcc61364d4c8ca12f0a116e827a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32879 } }
I20260924 19:33:07.063437 11630 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf00dcc61364d4c8ca12f0a116e827a; no voters:
I20260924 19:33:07.063588 11630 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:33:07.063642 11633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:33:07.063832 11630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:33:07.063854 11633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [term 1 LEADER]: Becoming Leader. State: Replica: 6bf00dcc61364d4c8ca12f0a116e827a, State: Running, Role: LEADER
I20260924 19:33:07.064010 11633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf00dcc61364d4c8ca12f0a116e827a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32879 } }
I20260924 19:33:07.064369 11634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf00dcc61364d4c8ca12f0a116e827a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf00dcc61364d4c8ca12f0a116e827a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32879 } } }
I20260924 19:33:07.064435 11634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:07.064658 11635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf00dcc61364d4c8ca12f0a116e827a. Latest consensus state: current_term: 1 leader_uuid: "6bf00dcc61364d4c8ca12f0a116e827a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf00dcc61364d4c8ca12f0a116e827a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32879 } } }
I20260924 19:33:07.064761 11635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:07.065037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 11562
I20260924 19:33:07.065129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:33:07.065296 392 external_mini_cluster.cc:1428] Setting key 6444195e7f7bb90584c364e4842e66d3
I20260924 19:33:07.065949 11630 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:33:07.066422 11640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:33:07.066866 11640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:33:07.077246 11640 catalog_manager.cc:1347] Generated new cluster ID: bdf8f4c576064b858142e355ff591e4e
I20260924 19:33:07.077334 11640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:33:07 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:07.085196 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:07.065745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.98.62:32879)
0924 19:33:07.065924 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:07.065935 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:07.065967 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:33:07.066087 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:07.066095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:07.066151 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:07.066312 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:07.068717 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:07.071350 (+ 2633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:07.077327 (+ 5977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:07.077996 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:07.078142 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:07.081052 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:07.081068 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:07.081069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:07.081082 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:07.083090 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:07.083463 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:07.083466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:07.083466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:07.083566 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:07.083769 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:07.083770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:07.083771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:07.084128 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:07.084276 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:07.084813 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 19:33:07.084940 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260924 19:33:07.090884 11640 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:33:07.091671 11640 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:33:07.098887 11640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a: Generated new TSK 0
I20260924 19:33:07.099146 11640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:33:07.840206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32879
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:07.923825 11675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:07.924042 11675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:07.924070 11675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:07.925220 11675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:07.925261 11675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:07.925287 11675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:33:07.926699 11675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:07.926899 11675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:07.927107 11675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:07.932818 11683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:07.932835 11681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:07.933080 11680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:07.933928 11675 server_base.cc:1034] running on GCE node
I20260924 19:33:07.934129 11675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:07.934386 11675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:07.935534 11675 hybrid_clock.cc:648] HybridClock initialized: now 1790278387935520 us; error 31 us; skew 500 ppm
Sep 24 19:33:07 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:07.938197 11675 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:33:07.938802 11675 webserver.cc:460] Webserver started at http://127.0.98.1:35369/ using document root <none> and password file <none>
I20260924 19:33:07.939018 11675 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:07.939072 11675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:07.939204 11675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:33:07 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:07.953590 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "b13ec9a304b54263a11d921b28447ed6"
format_stamp: "Formatted at 2026-09-24 19:33:07 on dist-test-slave-m8jb"
server_key: "2f116acd4becd3655f9a07d70a39d6d2"
server_key_iv: "29c9cd41b3daf9e6ff2a2a1f7fffa52b"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:07.954164 11675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b13ec9a304b54263a11d921b28447ed6"
format_stamp: "Formatted at 2026-09-24 19:33:07 on dist-test-slave-m8jb"
server_key: "2f116acd4becd3655f9a07d70a39d6d2"
server_key_iv: "29c9cd41b3daf9e6ff2a2a1f7fffa52b"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:07.961051 11675 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.000s
I20260924 19:33:07.980362 11690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:07.980654 11675 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260924 19:33:07.980778 11675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "b13ec9a304b54263a11d921b28447ed6"
format_stamp: "Formatted at 2026-09-24 19:33:07 on dist-test-slave-m8jb"
server_key: "2f116acd4becd3655f9a07d70a39d6d2"
server_key_iv: "29c9cd41b3daf9e6ff2a2a1f7fffa52b"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:07.980897 11675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:07.996268 11675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:07.998821 11675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:07.999081 11675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:07.999401 11675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:07.999848 11675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:07.999949 11675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:08.000048 11675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:08.000140 11675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:08.007249 11675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39997
I20260924 19:33:08.007530 11805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39997 every 8 connection(s)
I20260924 19:33:08.007869 11675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:33:08.017289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 11675
I20260924 19:33:08.017402 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:33:08.017586 392 external_mini_cluster.cc:1428] Setting key 053b40e761c6f94f75b02dfd2013fcf8
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 19:33:08.030793 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.009428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32879 (local address 127.0.98.1:46473)
0924 19:33:08.013483 (+ 4055us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.013549 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:08.014224 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:08.014707 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:08.014712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:08.014981 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:08.015294 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.015302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.016317 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.016319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.017678 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.017681 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.018465 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.018467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.018614 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.019263 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:08.019274 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:08.021131 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:08.026154 (+ 5023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:08.026168 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:08.026172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:08.029061 (+ 2889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:08.029398 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:08.029399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:08.029399 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:08.029586 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:08.029962 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:08.029971 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:08.030207 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 19:33:08.030329 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 19:33:08.030522 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3994,"thread_start_us":89,"threads_started":1}
I20260924 19:33:08.031286 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.009747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46473 (local address 127.0.98.62:32879)
0924 19:33:08.014445 (+ 4698us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.014447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.014456 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:33:08.014482 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.014483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.014498 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.014561 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:08.015416 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.016208 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.017780 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.018352 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.018426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.021258 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:08.021260 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:08.021260 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:08.021269 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:08.026036 (+ 4767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:08.029215 (+ 3179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:08.029217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:08.029217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:08.029310 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:08.029688 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:08.029689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:08.029690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:08.029859 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:08.029929 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.031065 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.031183 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4663,"thread_start_us":77,"threads_started":1}
I20260924 19:33:08.031648 11806 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32879
I20260924 19:33:08.031754 11806 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:08.031961 11806 heartbeater.cc:511] Master 127.0.98.62:32879 requested a full tablet report, sending...
I20260924 19:33:08.032524 11594 ts_manager.cc:194] Registered new tserver with Master: b13ec9a304b54263a11d921b28447ed6 (127.0.98.1:39997)
I20260924 19:33:08.033454 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:33:08.049001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32879
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:08.168080 11812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:08.168372 11812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:08.168411 11812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:08.170192 11812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:08.170249 11812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:08.170288 11812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:33:08.172441 11812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:08.172713 11812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:08.173022 11812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:33:08.179159 11819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:08.179155 11822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:08.179590 11820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:08.180326 11812 server_base.cc:1034] running on GCE node
I20260924 19:33:08.180550 11812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:08.180857 11812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:08.182029 11812 hybrid_clock.cc:648] HybridClock initialized: now 1790278388182007 us; error 32 us; skew 500 ppm
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:08.186991 11812 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:33:08.187618 11812 webserver.cc:460] Webserver started at http://127.0.98.2:41019/ using document root <none> and password file <none>
I20260924 19:33:08.187929 11812 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:08.188028 11812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:08.188186 11812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:08.216532 11812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25"
format_stamp: "Formatted at 2026-09-24 19:33:08 on dist-test-slave-m8jb"
server_key: "6b75f5ed579bd402f123c3f51b610ba0"
server_key_iv: "13cc80576c7e0f5f7890edf689058835"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:08.217044 11812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25"
format_stamp: "Formatted at 2026-09-24 19:33:08 on dist-test-slave-m8jb"
server_key: "6b75f5ed579bd402f123c3f51b610ba0"
server_key_iv: "13cc80576c7e0f5f7890edf689058835"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:08.225045 11812 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.000s
I20260924 19:33:08.237366 11829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:08.237540 11812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:33:08.237630 11812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25"
format_stamp: "Formatted at 2026-09-24 19:33:08 on dist-test-slave-m8jb"
server_key: "6b75f5ed579bd402f123c3f51b610ba0"
server_key_iv: "13cc80576c7e0f5f7890edf689058835"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:08.237725 11812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:08.273299 11812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:08.275602 11812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:08.275806 11812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:08.276077 11812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:08.276521 11812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:08.276616 11812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:08.276721 11812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:08.276774 11812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:08.283180 11812 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42591
I20260924 19:33:08.283649 11812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:33:08.285641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 11812
I20260924 19:33:08.285758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:33:08.285964 392 external_mini_cluster.cc:1428] Setting key 415fdfc77db1fe28db09e9df314b218a
I20260924 19:33:08.286528 11943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42591 every 8 connection(s)
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:08.297681 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.288291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45687 (local address 127.0.98.62:32879)
0924 19:33:08.288358 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.288360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.288369 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:33:08.288392 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.288393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.288401 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.288469 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:08.289324 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.290079 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.291523 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.291835 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.291881 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.294412 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:08.294413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:08.294414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:08.294433 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:08.295989 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:08.296494 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:08.296495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:08.296495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:08.296558 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:08.296883 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:08.296884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:08.296884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:08.297021 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:08.297066 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.297473 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.297564 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:33:08.298424 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.287079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32879 (local address 127.0.98.2:45687)
0924 19:33:08.287236 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.287281 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:08.287853 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:08.288562 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:08.288567 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:08.288824 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:08.289100 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.289108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.290194 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.290195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.291424 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.291426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.291972 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.291972 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.292112 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.292680 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:08.292690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:08.294305 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:08.296082 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:08.296089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:08.296091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:08.296403 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:08.296663 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:08.296664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:08.296664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:08.296797 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:08.297115 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:08.297121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:08.297311 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 19:33:08.298022 (+ 711us) client_negotiation.cc:240] Negotiation successful
0924 19:33:08.298184 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260924 19:33:08.299108 11944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32879
I20260924 19:33:08.299199 11944 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:08.299398 11944 heartbeater.cc:511] Master 127.0.98.62:32879 requested a full tablet report, sending...
I20260924 19:33:08.299750 11594 ts_manager.cc:194] Registered new tserver with Master: 8ee809f60fcc4a0e80ae3bb1106dbc25 (127.0.98.2:42591)
I20260924 19:33:08.300179 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45687
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:33:08.332161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33585/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32879
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:08.452172 11949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:08.452553 11949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:08.452677 11949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:08.454514 11949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:08.454751 11949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:08.454849 11949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:33:08.457252 11949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33585/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:08.457715 11949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:08.458047 11949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:08.464599 11957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:08.464589 11955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:08.464874 11954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:08.465699 11949 server_base.cc:1034] running on GCE node
I20260924 19:33:08.466012 11949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:08.466331 11949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:08.467505 11949 hybrid_clock.cc:648] HybridClock initialized: now 1790278388467488 us; error 38 us; skew 500 ppm
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:08.471829 11949 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:33:08.472396 11949 webserver.cc:460] Webserver started at http://127.0.98.3:38701/ using document root <none> and password file <none>
I20260924 19:33:08.473083 11949 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:08.473304 11949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:08.473471 11949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:08.492782 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "c79569bfb0ed4adbbdf85ab33d6589c0"
format_stamp: "Formatted at 2026-09-24 19:33:08 on dist-test-slave-m8jb"
server_key: "379ea6798817db2847b0f6a8bfcffcc7"
server_key_iv: "745d750962c074962f672ed1f95b714c"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:08.493247 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c79569bfb0ed4adbbdf85ab33d6589c0"
format_stamp: "Formatted at 2026-09-24 19:33:08 on dist-test-slave-m8jb"
server_key: "379ea6798817db2847b0f6a8bfcffcc7"
server_key_iv: "745d750962c074962f672ed1f95b714c"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:08.495173 11949 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:33:08.508756 11964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:08.508972 11949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:33:08.509064 11949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "c79569bfb0ed4adbbdf85ab33d6589c0"
format_stamp: "Formatted at 2026-09-24 19:33:08 on dist-test-slave-m8jb"
server_key: "379ea6798817db2847b0f6a8bfcffcc7"
server_key_iv: "745d750962c074962f672ed1f95b714c"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:08.509168 11949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:08.519778 11949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:08.521591 11949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:08.521718 11949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:08.521924 11949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:08.532724 11949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:08.532795 11949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:08.532825 11949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:08.532845 11949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:08.539341 11949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42217
I20260924 19:33:08.539769 11949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:33:08.544888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 11949
I20260924 19:33:08.545006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:33:08.545193 392 external_mini_cluster.cc:1428] Setting key 1db48c53a23df1026d9adc8295e5d6ed
I20260924 19:33:08.550423 12077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42217 every 8 connection(s)
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:08.561947 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.551405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55381 (local address 127.0.98.62:32879)
0924 19:33:08.551448 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.551450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.551658 (+ 208us) server_negotiation.cc:407] Connection header received
0924 19:33:08.552367 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.552369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.552382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.552412 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:08.553331 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.554101 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.555545 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.555855 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.555900 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.558695 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:08.558697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:08.558697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:08.558705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:08.560365 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:08.560884 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:08.560885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:08.560886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:08.560946 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:08.561217 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:08.561218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:08.561218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:08.561352 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:08.561402 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.561771 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.561851 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:33:08.562505 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.551104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32879 (local address 127.0.98.3:55381)
0924 19:33:08.551528 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.551579 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:08.552236 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:08.552649 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:08.552654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:08.552928 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:08.553223 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.553230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.554215 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.554216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.555447 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.555450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.555993 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.555993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.556139 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.556981 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:08.556992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:08.558587 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:08.560458 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:08.560464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:08.560466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:08.560794 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:08.561015 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:08.561016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:08.561016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:08.561146 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:08.561433 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:08.561439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:08.561612 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 19:33:08.562081 (+ 469us) client_negotiation.cc:240] Negotiation successful
0924 19:33:08.562263 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":96,"threads_started":1}
I20260924 19:33:08.562942 12078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32879
I20260924 19:33:08.563048 12078 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:08.563249 12078 heartbeater.cc:511] Master 127.0.98.62:32879 requested a full tablet report, sending...
I20260924 19:33:08.565497 11594 ts_manager.cc:194] Registered new tserver with Master: c79569bfb0ed4adbbdf85ab33d6589c0 (127.0.98.3:42217)
I20260924 19:33:08.565994 11594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55381
I20260924 19:33:08.567552 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:33:08.576967 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.569253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.0.98.62:32879)
0924 19:33:08.569291 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.569293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.569574 (+ 281us) server_negotiation.cc:407] Connection header received
0924 19:33:08.569661 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.569662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.569671 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.569697 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:08.570450 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.571201 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.572263 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.572522 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.572564 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.574126 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:08.574128 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:08.574128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:08.574136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:08.575707 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:08.576106 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:08.576108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:08.576108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:08.576164 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:08.576356 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:08.576358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:08.576358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:08.576477 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:08.576549 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.576795 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.576873 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:33:08.579037 11594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:33:08.581409 11594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:33:08.597028 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.585123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42591 (local address 127.0.0.1:36420)
0924 19:33:08.586496 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.586540 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:08.587101 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:08.590603 (+ 3502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:08.590603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:08.590606 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:08.590657 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.590659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.591903 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.591904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.594807 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.594810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.596785 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.596786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.596848 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.596851 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:33:08.596898 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:33:08.596938 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1344,"thread_start_us":92,"threads_started":1}
I20260924 19:33:08.597560 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.585885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.0.98.2:42591)
0924 19:33:08.587250 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.587255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.587266 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:33:08.587306 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.587308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.587324 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.587392 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:08.590751 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.591808 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.594918 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.595674 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.595755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.595836 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.597438 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.597480 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1326,"thread_start_us":77,"threads_started":1}
I20260924 19:33:08.597886 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.586045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.0.98.3:42217)
0924 19:33:08.587527 (+ 1482us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.587531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.587807 (+ 276us) server_negotiation.cc:407] Connection header received
0924 19:33:08.587850 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.587853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.587868 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.587927 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:08.588448 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.589553 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.594157 (+ 4604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.596064 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.596128 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.596201 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.597770 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.597813 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1448,"thread_start_us":65,"threads_started":1}
I20260924 19:33:08.597373 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.585563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42217 (local address 127.0.0.1:40098)
0924 19:33:08.587699 (+ 2136us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.587707 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:08.587754 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:08.588014 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:08.588292 (+ 278us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:08.588296 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:08.588368 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.588370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.589648 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.589649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.594041 (+ 4392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.594044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.596237 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.596238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.596320 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.596334 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:33:08.596404 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:33:08.597304 (+ 900us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2124,"thread_start_us":60,"threads_started":1}
I20260924 19:33:08.599126 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 6a000ffc960341499b59a550a8f0904a (DEFAULT_TABLE table=test-table [id=d43ab8d31e1f4df1ba71598a79432035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:33:08.599195 12012 tablet_service.cc:1467] Processing CreateTablet for tablet 6a000ffc960341499b59a550a8f0904a (DEFAULT_TABLE table=test-table [id=d43ab8d31e1f4df1ba71598a79432035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:33:08.599548 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a000ffc960341499b59a550a8f0904a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:08.599560 12012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a000ffc960341499b59a550a8f0904a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:08.607543 12097 tablet_bootstrap.cc:492] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: Bootstrap starting.
I20260924 19:33:08.609074 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.585385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39997 (local address 127.0.0.1:43772)
0924 19:33:08.587578 (+ 2193us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.587588 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:08.587630 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:08.603914 (+ 16284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:08.603915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:08.603918 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:08.603966 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.603968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.605350 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.605351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.606465 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.606468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.608832 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.608833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.608905 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.608910 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:33:08.608941 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 19:33:08.608989 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2177,"thread_start_us":67,"threads_started":1}
I20260924 19:33:08.609652 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.585484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.0.98.1:39997)
0924 19:33:08.603724 (+ 18240us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.603729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.603755 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:33:08.603789 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.603791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.603808 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.604033 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:08.604098 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.605246 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.606570 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.607352 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.609424 (+ 2072us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.609506 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.609534 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.609580 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18199,"thread_start_us":57,"threads_started":1}
I20260924 19:33:08.610530 11726 tablet_service.cc:1467] Processing CreateTablet for tablet 6a000ffc960341499b59a550a8f0904a (DEFAULT_TABLE table=test-table [id=d43ab8d31e1f4df1ba71598a79432035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:33:08.610882 11726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a000ffc960341499b59a550a8f0904a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:08.610889 12097 tablet_bootstrap.cc:654] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:08.611243 12097 log.cc:824] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:08.612092 12097 tablet_bootstrap.cc:492] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: No bootstrap required, opened a new log
I20260924 19:33:08.612092 12096 tablet_bootstrap.cc:492] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25: Bootstrap starting.
I20260924 19:33:08.612162 12097 ts_tablet_manager.cc:1397] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 19:33:08.613121 12096 tablet_bootstrap.cc:654] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:08.613382 12096 log.cc:824] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:08.614121 12097 raft_consensus.cc:348] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.614283 12097 raft_consensus.cc:374] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:08.614316 12097 raft_consensus.cc:729] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79569bfb0ed4adbbdf85ab33d6589c0, State: Initialized, Role: FOLLOWER
I20260924 19:33:08.614428 12097 consensus_queue.cc:260] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.614679 12097 ts_tablet_manager.cc:1428] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:33:08.615574 12078 heartbeater.cc:503] Master 127.0.98.62:32879 was elected leader, sending a full tablet report...
I20260924 19:33:08.617158 12100 tablet_bootstrap.cc:492] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6: Bootstrap starting.
I20260924 19:33:08.617883 12100 tablet_bootstrap.cc:654] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:08.618247 12100 log.cc:824] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:08.619014 12100 tablet_bootstrap.cc:492] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6: No bootstrap required, opened a new log
I20260924 19:33:08.619081 12100 ts_tablet_manager.cc:1397] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:33:08.621284 12096 tablet_bootstrap.cc:492] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25: No bootstrap required, opened a new log
I20260924 19:33:08.621368 12096 ts_tablet_manager.cc:1397] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260924 19:33:08.621768 12100 raft_consensus.cc:348] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.621937 12100 raft_consensus.cc:374] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:08.621973 12100 raft_consensus.cc:729] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13ec9a304b54263a11d921b28447ed6, State: Initialized, Role: FOLLOWER
I20260924 19:33:08.622215 12100 consensus_queue.cc:260] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.622542 12100 ts_tablet_manager.cc:1428] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:33:08.622692 11806 heartbeater.cc:503] Master 127.0.98.62:32879 was elected leader, sending a full tablet report...
I20260924 19:33:08.623258 12096 raft_consensus.cc:348] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.623412 12096 raft_consensus.cc:374] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:08.623440 12096 raft_consensus.cc:729] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee809f60fcc4a0e80ae3bb1106dbc25, State: Initialized, Role: FOLLOWER
I20260924 19:33:08.623543 12096 consensus_queue.cc:260] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.623782 12096 ts_tablet_manager.cc:1428] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 19:33:08.623888 11944 heartbeater.cc:503] Master 127.0.98.62:32879 was elected leader, sending a full tablet report...
Sep 24 19:33:08 dist-test-slave-m8jb krb5kdc[11118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:08.720410 12101 raft_consensus.cc:482] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:33:08.720562 12101 raft_consensus.cc:504] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.721009 12101 leader_election.cc:288] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee809f60fcc4a0e80ae3bb1106dbc25 (127.0.98.2:42591), b13ec9a304b54263a11d921b28447ed6 (127.0.98.1:39997)
I20260924 19:33:08.725282 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.721355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42591 (local address 127.0.98.3:34065)
0924 19:33:08.721684 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.721696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:08.721749 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:08.722167 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:08.722168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:08.722171 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:08.722244 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.722246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.723115 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.723116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.724251 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.724255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.725042 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.725043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.725117 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.725121 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:33:08.725166 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:33:08.725208 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308}
I20260924 19:33:08.725761 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.721874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34065 (local address 127.0.98.2:42591)
0924 19:33:08.721971 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.721973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.721982 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:33:08.722010 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.722011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.722023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.722070 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:08.722323 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.723003 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.724353 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.724926 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.724997 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.725010 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.725641 (+ 631us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.725682 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:33:08.726037 11898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a000ffc960341499b59a550a8f0904a" candidate_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" is_pre_election: true
I20260924 19:33:08.726220 11898 raft_consensus.cc:2454] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c79569bfb0ed4adbbdf85ab33d6589c0 in term 0.
I20260924 19:33:08.726454 11965 leader_election.cc:302] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee809f60fcc4a0e80ae3bb1106dbc25, c79569bfb0ed4adbbdf85ab33d6589c0; no voters:
I20260924 19:33:08.726577 12101 raft_consensus.cc:2781] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:33:08.726614 12101 raft_consensus.cc:482] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:33:08.726634 12101 raft_consensus.cc:3037] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:08.727515 12101 raft_consensus.cc:504] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.727653 12101 leader_election.cc:288] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee809f60fcc4a0e80ae3bb1106dbc25 (127.0.98.2:42591), b13ec9a304b54263a11d921b28447ed6 (127.0.98.1:39997)
I20260924 19:33:08.727833 11898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a000ffc960341499b59a550a8f0904a" candidate_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25"
I20260924 19:33:08.727919 11898 raft_consensus.cc:3037] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:08.728847 11898 raft_consensus.cc:2454] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79569bfb0ed4adbbdf85ab33d6589c0 in term 1.
I20260924 19:33:08.729010 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.721459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39997 (local address 127.0.98.3:47423)
0924 19:33:08.725353 (+ 3894us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.725363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:08.725400 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:08.725922 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:08.725923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:08.725925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:08.725956 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.725957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.726829 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.726830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.727777 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:08.727780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.728809 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.728810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:08.728864 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.728867 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:33:08.728913 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:33:08.728947 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3883,"thread_start_us":112,"threads_started":1}
I20260924 19:33:08.729209 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.721928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47423 (local address 127.0.98.1:39997)
0924 19:33:08.722132 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.722134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.725448 (+ 3314us) server_negotiation.cc:407] Connection header received
0924 19:33:08.725476 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.725477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.725489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.725528 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:08.726030 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.726725 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.728094 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.728684 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.728764 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.728776 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.729103 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.729149 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260924 19:33:08.729193 11965 leader_election.cc:302] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee809f60fcc4a0e80ae3bb1106dbc25, c79569bfb0ed4adbbdf85ab33d6589c0; no voters:
I20260924 19:33:08.729329 12101 raft_consensus.cc:2781] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:33:08.729467 12101 raft_consensus.cc:686] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 1 LEADER]: Becoming Leader. State: Replica: c79569bfb0ed4adbbdf85ab33d6589c0, State: Running, Role: LEADER
I20260924 19:33:08.729477 11754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a000ffc960341499b59a550a8f0904a" candidate_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13ec9a304b54263a11d921b28447ed6" is_pre_election: true
I20260924 19:33:08.729552 12101 consensus_queue.cc:237] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } }
I20260924 19:33:08.729610 11754 raft_consensus.cc:2454] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c79569bfb0ed4adbbdf85ab33d6589c0 in term 0.
I20260924 19:33:08.730000 11745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a000ffc960341499b59a550a8f0904a" candidate_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b13ec9a304b54263a11d921b28447ed6"
I20260924 19:33:08.730087 11745 raft_consensus.cc:3037] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:08.730363 11594 catalog_manager.cc:5495] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c79569bfb0ed4adbbdf85ab33d6589c0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c79569bfb0ed4adbbdf85ab33d6589c0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42217 } health_report { overall_health: HEALTHY } } }
I20260924 19:33:08.730856 11745 raft_consensus.cc:2454] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c79569bfb0ed4adbbdf85ab33d6589c0 in term 1.
I20260924 19:33:08.748953 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.745683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.0.98.3:42217)
0924 19:33:08.745719 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:08.745721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:08.745727 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:33:08.745775 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:08.745776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:08.745797 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:08.745851 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:33:08.746039 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.746752 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.747766 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:08.748067 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:08.748128 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:08.748354 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:33:08.748487 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:33:08.748566 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:08.748831 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:33:08.748851 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:33:08.757424 11745 raft_consensus.cc:1261] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6 [term 1 FOLLOWER]: Refusing update from remote peer c79569bfb0ed4adbbdf85ab33d6589c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:33:08.757757 11898 raft_consensus.cc:1261] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 1 FOLLOWER]: Refusing update from remote peer c79569bfb0ed4adbbdf85ab33d6589c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:33:08.758108 12106 consensus_queue.cc:1035] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee809f60fcc4a0e80ae3bb1106dbc25" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:33:08.758208 12101 consensus_queue.cc:1035] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b13ec9a304b54263a11d921b28447ed6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:33:08.759644 11807 tablet.cc:2369] T 6a000ffc960341499b59a550a8f0904a P b13ec9a304b54263a11d921b28447ed6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:33:08.767388 12109 mvcc.cc:205] Tried to move back new op lower bound from 7332980280346550272 to 7332980280236859392. Current Snapshot: MvccSnapshot[applied={T|T < 7332980280346550272}]
I20260924 19:33:08.768707 12111 mvcc.cc:205] Tried to move back new op lower bound from 7332980280346550272 to 7332980280236859392. Current Snapshot: MvccSnapshot[applied={T|T < 7332980280346550272}]
I20260924 19:33:08.772720 12110 mvcc.cc:205] Tried to move back new op lower bound from 7332980280346550272 to 7332980280236859392. Current Snapshot: MvccSnapshot[applied={T|T < 7332980280346550272}]
I20260924 19:33:08.775743 11594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:33:08.775838 11594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:33:08.776695 11594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bf00dcc61364d4c8ca12f0a116e827a: Sending DeleteTablet for 3 replicas of tablet 6a000ffc960341499b59a550a8f0904a
I20260924 19:33:08.777195 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a000ffc960341499b59a550a8f0904a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:33:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36420
I20260924 19:33:08.777408 11726 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a000ffc960341499b59a550a8f0904a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:33:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43772
I20260924 19:33:08.777601 12012 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a000ffc960341499b59a550a8f0904a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:33:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40098
I20260924 19:33:08.778262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 11675
I20260924 19:33:08.781844 12120 tablet_replica.cc:330] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: stopping tablet replica
I20260924 19:33:08.781982 12120 raft_consensus.cc:2229] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:33:08.782186 12120 raft_consensus.cc:2258] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:33:08.782724 12120 ts_tablet_manager.cc:1905] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:33:08.785200 12120 ts_tablet_manager.cc:1918] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:33:08.787375 12118 tablet_replica.cc:330] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25: stopping tablet replica
I20260924 19:33:08.787472 12118 raft_consensus.cc:2229] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:33:08.787581 12118 raft_consensus.cc:2258] T 6a000ffc960341499b59a550a8f0904a P 8ee809f60fcc4a0e80ae3bb1106dbc25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:33:08.787580 12120 log.cc:1196] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/6a000ffc960341499b59a550a8f0904a
I20260924 19:33:08.787937 12120 ts_tablet_manager.cc:1939] T 6a000ffc960341499b59a550a8f0904a P c79569bfb0ed4adbbdf85ab33d6589c0: Deleting consensus metadata
I20260924 19:33:08.788430 11580 catalog_manager.cc:4841] TS c79569bfb0ed4adbbdf85ab33d6589c0 (127.0.98.3:42217): tablet 6a000ffc960341499b59a550a8f0904a (table test-table [id=d43ab8d31e1f4df1ba71598a79432035]) successfully deleted
W20260924 19:33:08.789942 11578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39997 (error 108)
I20260924 19:33:08.790827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 11812
I20260924 19:33:08.797036 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.796912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39997 (local address 127.0.0.1:43786)
0924 19:33:08.796958 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.796997 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 19:33:08.797163 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:08.797100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42591 (local address 127.0.0.1:36424)
0924 19:33:08.797124 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:08.797145 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 19:33:08.797178 11578 catalog_manager.cc:4584] TS b13ec9a304b54263a11d921b28447ed6 (127.0.98.1:39997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a000ffc960341499b59a550a8f0904a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39997: connect: Connection refused (error 111)
I20260924 19:33:08.797519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 11949
I20260924 19:33:08.803259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 11562
I20260924 19:33:08.813342 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:33:09.147512 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:33:09.147603 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:33:09.492337 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:33:09.492 UTC [11148] LOG: received smart shutdown request
2026-09-24 19:33:09.495 UTC [11148] LOG: background worker "logical replication launcher" (PID 11157) exited with exit code 1
2026-09-24 19:33:09.496 UTC [11152] LOG: shutting down
2026-09-24 19:33:09.659 UTC [11148] LOG: database system is shut down
2026-09-24T19:33:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44115 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:33:09 dist-test-slave-m8jb krb5kdc[12130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:33:09 dist-test-slave-m8jb krb5kdc[12130](info): set up 2 sockets
Sep 24 19:33:09 dist-test-slave-m8jb krb5kdc[12130](info): commencing operation
krb5kdc: starting...
W20260924 19:33:10.696290 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:33:10 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:33:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:33:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 19:33:10.855113 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLXPSqI/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/postgres -l logfile start
2026-09-24 19:33:13.363 UTC [12160] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:33:13.363 UTC [12160] LOG: listening on IPv4 address "127.0.98.20", port 36283
2026-09-24 19:33:13.367 UTC [12160] LOG: listening on Unix socket "/tmp/.s.PGSQL.36283"
2026-09-24 19:33:13.388 UTC [12163] LOG: database system was shut down at 2026-09-24 19:33:11 UTC
2026-09-24 19:33:13.391 UTC [12160] LOG: database system is ready to accept connections
I20260924 19:33:14.252250 392 mini_postgres.cc:96] Postgres bound to 36283
2026-09-24 19:33:14.257 UTC [12171] FATAL: database "slave" does not exist
127.0.98.20:36283 - accepting connections
I20260924 19:33:14.258301 392 mini_ranger.cc:161] Starting Ranger...
I20260924 19:33:14.272531 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:33:14.747182 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:33:14.747262 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-admin
2026-09-24 19:33:15,006 [I] DB FLAVOR :POSTGRES
2026-09-24 19:33:15,007 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:33:15,007 [I] Checking connection..
2026-09-24 19:33:15,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:33:15,262 [I] Checking connection passed.
2026-09-24 19:33:15,262 [I] --------- Verifying version history table ---------
2026-09-24 19:33:15,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:33:15,501 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:33:15,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:33:15,745 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:33:15,746 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:33:15,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:33:15.971 UTC [12235] WARNING: there is no transaction in progress
2026-09-24 19:33:15,976 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:33:15,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:33:16,220 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:33:16,220 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:33:16,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:33:16,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:33:16,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m8jb.c.gcp-upstream.internal','N') ;"
2026-09-24 19:33:16,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:33:17,175 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:33:17,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:33:17,426 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:33:17,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:33:17,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:33:17,925 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:33:17,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:33:18.870 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:18.886 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:18.902 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:18.917 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:18.932 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:18.953 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.008 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.015 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.030 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.041 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.051 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.065 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.074 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.080 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.087 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19.094 UTC [12343] WARNING: there is no transaction in progress
2026-09-24 19:33:19,737 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:33:19,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:33:19,982 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:33:19,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:33:20,237 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:33:20,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:33:20,472 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:33:20,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:33:20,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:33:20,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m8jb.c.gcp-upstream.internal';"
2026-09-24 19:33:21,215 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:33:21,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:33:21,446 [I] DB_PATCHES have already been applied
I20260924 19:33:21.451721 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLXPSqI/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:33:21.451792 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 19:33:21.454609 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:48557
Sep 24, 2026 7:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:33:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:33:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:33:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:33:22.454650 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.196s user 0.001s sys 0.007s
Sep 24, 2026 7:33:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:33:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:33:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:33:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:33:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:33:42.567380 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:33:42 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278422, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:33:42.644721 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:33:42.653795 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:33:43.564160 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:33:43.564232 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms
I20260924 19:33:43.564244 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:36283
2026-09-24 19:33:43,710 [I] DB FLAVOR :POSTGRES
2026-09-24 19:33:43,711 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:33:43,711 [I] Checking connection
2026-09-24 19:33:43,711 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:33:43,966 [I] connection success
2026-09-24 19:33:43,966 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:33:43,966 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:33:43,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:33:44,220 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:33:44,221 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:33:44,221 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:33:44,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36283/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:33:44,470 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:33:44.911376 392 mini_ranger_kms.cc:310] Created kms service
I20260924 19:33:45.070096 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:33:45.113837 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:33:45 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278422, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:45.494799 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:33:45.494917 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLXPSqI/build/release/bin/postgresql.jar:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLXPSqI/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:33:45.494939 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 19:33:45.498405 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12523
I20260924 19:33:45.498483 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58247
Sep 24, 2026 7:33:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:33:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:33:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:33:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:33:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:33:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:33:47.530016 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.885s user 0.008s sys 0.121s
I20260924 19:33:47.530107 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:33:47.530150 392 mini_ranger_kms.cc:401] 127.0.98.20:58247/kms/v1/keys
Sep 24 19:33:47 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:33:48 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 19:33:48 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278427, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 19:33:50 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:33:50.800149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40547
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40547
--ranger_config_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58247/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:50.909925 12574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:50.910143 12574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:50.910164 12574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:50.911322 12574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:33:50.911366 12574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:50.911378 12574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:50.911389 12574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:33:50.911401 12574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:33:50.913035 12574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58247/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40547
--ranger_config_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:50.913282 12574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:50.913487 12574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:50.918334 12579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:50.918363 12580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:50.918357 12582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:50.918571 12574 server_base.cc:1034] running on GCE node
I20260924 19:33:50.918885 12574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:50.919163 12574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:50.920301 12574 hybrid_clock.cc:648] HybridClock initialized: now 1790278430920289 us; error 31 us; skew 500 ppm
Sep 24 19:33:50 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:50.927546 12574 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:33:50.928051 12574 webserver.cc:460] Webserver started at http://127.0.98.62:34895/ using document root <none> and password file <none>
I20260924 19:33:50.928273 12574 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:50.928315 12574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:50.928458 12574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:33:50 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:50.981484 12574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "3b995882abfd4007a2340922fd424ce0"
format_stamp: "Formatted at 2026-09-24 19:33:50 on dist-test-slave-m8jb"
server_key: "ffbfb4e2cecba1dd4e53907f5e41d303"
server_key_iv: "68d06773a6a4d50e1f40a84732d09ca9"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:50.981986 12574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b995882abfd4007a2340922fd424ce0"
format_stamp: "Formatted at 2026-09-24 19:33:50 on dist-test-slave-m8jb"
server_key: "ffbfb4e2cecba1dd4e53907f5e41d303"
server_key_iv: "68d06773a6a4d50e1f40a84732d09ca9"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:50.983811 12574 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:33:50.995862 12589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:50.996182 12574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:33:50.996289 12574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "3b995882abfd4007a2340922fd424ce0"
format_stamp: "Formatted at 2026-09-24 19:33:50 on dist-test-slave-m8jb"
server_key: "ffbfb4e2cecba1dd4e53907f5e41d303"
server_key_iv: "68d06773a6a4d50e1f40a84732d09ca9"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:50.996382 12574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:51.016319 12574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:51.018807 12574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:51.018951 12574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:51.023473 12574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40547
I20260924 19:33:51.023536 12641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40547 every 8 connection(s)
I20260924 19:33:51.023885 12574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:33:51.024662 12642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:51.026826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 12574
I20260924 19:33:51.026922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:33:51.027124 392 external_mini_cluster.cc:1428] Setting key d5959ec8e4e18bf76479ba55746bf929
I20260924 19:33:51.028184 12642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Bootstrap starting.
I20260924 19:33:51.028942 12642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:51.029258 12642 log.cc:824] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:51.030043 12642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: No bootstrap required, opened a new log
I20260924 19:33:51.032450 12642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } }
I20260924 19:33:51.032625 12642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:51.032727 12642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b995882abfd4007a2340922fd424ce0, State: Initialized, Role: FOLLOWER
I20260924 19:33:51.032871 12642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } }
I20260924 19:33:51.032946 12642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:33:51.032996 12642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:33:51.033048 12642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 19:33:51 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:51.033712 12642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } }
I20260924 19:33:51.033823 12642 leader_election.cc:302] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b995882abfd4007a2340922fd424ce0; no voters:
I20260924 19:33:51.033977 12642 leader_election.cc:288] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:33:51.034024 12647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:33:51.034330 12647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 3b995882abfd4007a2340922fd424ce0, State: Running, Role: LEADER
I20260924 19:33:51.034397 12642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:33:51.034476 12647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } }
I20260924 19:33:51.034808 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b995882abfd4007a2340922fd424ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } } }
I20260924 19:33:51.034885 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:51.035007 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b995882abfd4007a2340922fd424ce0. Latest consensus state: current_term: 1 leader_uuid: "3b995882abfd4007a2340922fd424ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } } }
I20260924 19:33:51.035127 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:51.035297 12651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:33:51.035832 12651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:33:51.036590 12642 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:33:51.040743 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:51.027665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.0.98.62:40547)
0924 19:33:51.027854 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:51.027864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:51.027895 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:33:51.028313 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:51.028323 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:51.028392 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:51.028522 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:51.029196 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:51.030288 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:51.031061 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:51.031486 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:51.031578 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:51.033909 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:51.033922 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:51.033922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:51.033936 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:51.035866 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:51.036259 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:51.036262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:51.036262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:51.036353 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:51.036565 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:51.036566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:51.036566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:51.036882 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:51.039435 (+ 2553us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:51.039699 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 19:33:51.040296 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260924 19:33:51.042842 12651 catalog_manager.cc:1347] Generated new cluster ID: 5240428061e8451ea7f66e0d776b40d7
I20260924 19:33:51.042896 12651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:33:51.056713 12651 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:33:51.057580 12651 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:33:51.064296 12651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Generated new TSK 0
I20260924 19:33:51.064517 12651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:33:51.798919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40547
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:51.901392 12687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:51.901651 12687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:51.901685 12687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:51.903234 12687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:51.903285 12687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:51.903311 12687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:33:51.904760 12687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:51.904968 12687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:51.905174 12687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:51.909787 12692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:51.909794 12695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:51.910045 12693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:51.910089 12687 server_base.cc:1034] running on GCE node
I20260924 19:33:51.910279 12687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:51.910485 12687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:51.911633 12687 hybrid_clock.cc:648] HybridClock initialized: now 1790278431911617 us; error 32 us; skew 500 ppm
Sep 24 19:33:51 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:51.913914 12687 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:33:51.914332 12687 webserver.cc:460] Webserver started at http://127.0.98.1:36057/ using document root <none> and password file <none>
I20260924 19:33:51.914539 12687 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:51.914580 12687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:51.914690 12687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:33:51 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:51.926787 12687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "e27d4ad82bd846c0a380aac4aaea0c3d"
format_stamp: "Formatted at 2026-09-24 19:33:51 on dist-test-slave-m8jb"
server_key: "9f9bc1cc1952874938d44577bd7093d4"
server_key_iv: "d434242cc09e8793fee0ed8e5fb4060a"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:51.927198 12687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e27d4ad82bd846c0a380aac4aaea0c3d"
format_stamp: "Formatted at 2026-09-24 19:33:51 on dist-test-slave-m8jb"
server_key: "9f9bc1cc1952874938d44577bd7093d4"
server_key_iv: "d434242cc09e8793fee0ed8e5fb4060a"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:51.928761 12687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:33:51.942461 12703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:51.942698 12687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:33:51.942797 12687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "e27d4ad82bd846c0a380aac4aaea0c3d"
format_stamp: "Formatted at 2026-09-24 19:33:51 on dist-test-slave-m8jb"
server_key: "9f9bc1cc1952874938d44577bd7093d4"
server_key_iv: "d434242cc09e8793fee0ed8e5fb4060a"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:51.942891 12687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:51.991350 12687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:51.993232 12687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:51.993379 12687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:51.993574 12687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:51.993930 12687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:51.994020 12687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:51.994120 12687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:51.994194 12687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.000808 12687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37595
I20260924 19:33:52.001180 12687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:33:52.001466 12817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37595 every 8 connection(s)
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:52.008548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 12687
I20260924 19:33:52.008666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:33:52.008885 392 external_mini_cluster.cc:1428] Setting key b5b1ebe63378ad6312fe6f5d975ab9fe
I20260924 19:33:52.012087 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:52.001912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40547 (local address 127.0.98.1:44005)
0924 19:33:52.002058 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:52.002100 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:52.002622 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:52.002874 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:52.002878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:52.003090 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:52.003372 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:52.003379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.004384 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.004385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:52.005693 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:52.005696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.006254 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.006255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:52.006392 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:52.006992 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:52.007002 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:52.008612 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:52.010508 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:52.010515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:52.010518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:52.010794 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:52.011004 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:52.011004 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:52.011005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:52.011127 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:52.011410 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:52.011416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:52.011589 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 19:33:52.011691 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:33:52.011855 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260924 19:33:52.012449 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:52.001893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44005 (local address 127.0.98.62:40547)
0924 19:33:52.002303 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:52.002305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:52.002317 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:33:52.002722 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:52.002724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:52.002738 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:52.002802 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:52.003483 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.004272 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.005802 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.006113 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.006173 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:52.008732 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:52.008734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:52.008734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:52.008753 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:52.010425 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:52.010872 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:52.010874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:52.010874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:52.010941 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:52.011194 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:52.011195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:52.011195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:52.011327 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:52.011385 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:52.012274 (+ 889us) server_negotiation.cc:299] Negotiation successful
0924 19:33:52.012354 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":70,"threads_started":1}
I20260924 19:33:52.013042 12818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40547
I20260924 19:33:52.013139 12818 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:52.013346 12818 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:52.013876 12606 ts_manager.cc:194] Registered new tserver with Master: e27d4ad82bd846c0a380aac4aaea0c3d (127.0.98.1:37595)
I20260924 19:33:52.014720 12606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44005
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:33:52.061096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40547
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:33:52.183374 12826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:52.183763 12826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:52.183821 12826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:52.185591 12826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:52.185653 12826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:52.185693 12826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:33:52.187903 12826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:52.188166 12826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:52.188478 12826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:52.194592 12835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:52.194837 12832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:52.195283 12833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:52.195775 12826 server_base.cc:1034] running on GCE node
I20260924 19:33:52.196004 12826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:52.196283 12826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:52.197453 12826 hybrid_clock.cc:648] HybridClock initialized: now 1790278432197428 us; error 39 us; skew 500 ppm
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:52.200699 12826 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:33:52.201246 12826 webserver.cc:460] Webserver started at http://127.0.98.2:43333/ using document root <none> and password file <none>
I20260924 19:33:52.201493 12826 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:52.201551 12826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:52.201721 12826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:52.214453 12826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "66d9d85d573347c4b17243286e9e9427"
format_stamp: "Formatted at 2026-09-24 19:33:52 on dist-test-slave-m8jb"
server_key: "b5a3ae5af032f37f100e891362209720"
server_key_iv: "7bd4dbee60aee7e664d693fa03c466c6"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:52.214929 12826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "66d9d85d573347c4b17243286e9e9427"
format_stamp: "Formatted at 2026-09-24 19:33:52 on dist-test-slave-m8jb"
server_key: "b5a3ae5af032f37f100e891362209720"
server_key_iv: "7bd4dbee60aee7e664d693fa03c466c6"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:52.216903 12826 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:33:52.240816 12842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.241098 12826 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260924 19:33:52.241250 12826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "66d9d85d573347c4b17243286e9e9427"
format_stamp: "Formatted at 2026-09-24 19:33:52 on dist-test-slave-m8jb"
server_key: "b5a3ae5af032f37f100e891362209720"
server_key_iv: "7bd4dbee60aee7e664d693fa03c466c6"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:52.241428 12826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:52.263813 12826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:52.266268 12826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:52.266520 12826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:52.266781 12826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:52.267191 12826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:52.267277 12826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.267339 12826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:52.267390 12826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.274221 12826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45223
I20260924 19:33:52.274720 12826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:33:52.276818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 12826
I20260924 19:33:52.276907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:33:52.277096 392 external_mini_cluster.cc:1428] Setting key 9f898470da18d9553a24a339480abd0a
I20260924 19:33:52.277973 12955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45223 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:52.302218 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:52.277630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40547 (local address 127.0.98.2:51953)
0924 19:33:52.278479 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:52.278524 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:52.279152 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:52.280051 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:52.280056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:52.280315 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:52.280598 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:52.280604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.292802 (+ 12198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.292804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:52.293815 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:52.293820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.295997 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.295999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:52.296189 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:52.296855 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:52.296868 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:52.298488 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:52.300405 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:52.300413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:52.300416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:52.300746 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:52.300985 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:52.300986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:52.300986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:52.301129 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:52.301488 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:52.301495 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:52.301684 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 19:33:52.301790 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:33:52.301957 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":86,"threads_started":1}
I20260924 19:33:52.302652 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:52.277935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51953 (local address 127.0.98.62:40547)
0924 19:33:52.277978 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:52.277980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:52.279474 (+ 1494us) server_negotiation.cc:407] Connection header received
0924 19:33:52.279505 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:52.279506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:52.279518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:52.279579 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:52.280751 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.281484 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.295252 (+ 13768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.295575 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.295674 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:52.298603 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:52.298605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:52.298605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:52.298614 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:52.300282 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:52.300828 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:52.300830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:52.300830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:52.300902 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:52.301209 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:52.301210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:52.301210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:52.301351 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:52.301412 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:52.302479 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0924 19:33:52.302568 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:33:52.302997 12956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40547
I20260924 19:33:52.303094 12956 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:52.303305 12956 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:52.303763 12606 ts_manager.cc:194] Registered new tserver with Master: 66d9d85d573347c4b17243286e9e9427 (127.0.98.2:45223)
I20260924 19:33:52.304272 12606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:33:52.317943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40547
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:52.425800 12961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:52.426007 12961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:52.426036 12961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:52.427156 12961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:52.427196 12961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:52.427228 12961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:33:52.429044 12961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:52.429406 12961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:52.429649 12961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:52.434633 12967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:52.435113 12969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:52.435437 12966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:52.436705 12961 server_base.cc:1034] running on GCE node
I20260924 19:33:52.437017 12961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:52.437287 12961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:52.438441 12961 hybrid_clock.cc:648] HybridClock initialized: now 1790278432438418 us; error 36 us; skew 500 ppm
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:52.440845 12961 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:33:52.441335 12961 webserver.cc:460] Webserver started at http://127.0.98.3:38045/ using document root <none> and password file <none>
I20260924 19:33:52.441552 12961 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:52.441598 12961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:52.441725 12961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:52.454181 12961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "13c1ff28098b4244b97697bd84dde03c"
format_stamp: "Formatted at 2026-09-24 19:33:52 on dist-test-slave-m8jb"
server_key: "3c8538dad55f86d8148d15033c46ace6"
server_key_iv: "b24b289d845d2b2827fa3f7cd179af15"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:52.454691 12961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "13c1ff28098b4244b97697bd84dde03c"
format_stamp: "Formatted at 2026-09-24 19:33:52 on dist-test-slave-m8jb"
server_key: "3c8538dad55f86d8148d15033c46ace6"
server_key_iv: "b24b289d845d2b2827fa3f7cd179af15"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:52.465008 12961 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260924 19:33:52.485934 12976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.486208 12961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:33:52.486315 12961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "13c1ff28098b4244b97697bd84dde03c"
format_stamp: "Formatted at 2026-09-24 19:33:52 on dist-test-slave-m8jb"
server_key: "3c8538dad55f86d8148d15033c46ace6"
server_key_iv: "b24b289d845d2b2827fa3f7cd179af15"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:52.486418 12961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:52.515035 12961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:52.517634 12961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:52.517832 12961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:52.518080 12961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:52.518478 12961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:52.518564 12961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.518631 12961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:52.518684 12961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.525234 12961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33789
I20260924 19:33:52.525712 12961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:33:52.528791 13089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33789 every 8 connection(s)
I20260924 19:33:52.533403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 12961
I20260924 19:33:52.533501 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:33:52.533705 392 external_mini_cluster.cc:1428] Setting key 16af12f0ff75acf23ea73f29166c86cc
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:52.541085 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:52.528439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46485 (local address 127.0.98.62:40547)
0924 19:33:52.528498 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:52.528500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:52.532883 (+ 4383us) server_negotiation.cc:407] Connection header received
0924 19:33:52.533514 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:52.533516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:52.533530 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:52.533561 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:52.534509 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.535320 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.536475 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.536707 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.536765 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:52.538647 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:52.538649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:52.538649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:52.538657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:52.539868 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:52.540210 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:52.540211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:52.540212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:52.540263 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:52.540472 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:52.540473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:52.540474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:52.540575 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:52.540648 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:52.540923 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 19:33:52.540997 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:33:52.541308 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:52.528147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40547 (local address 127.0.98.3:46485)
0924 19:33:52.532743 (+ 4596us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:52.532789 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:52.533381 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:52.533778 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:52.533783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:52.534077 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:52.534367 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:52.534375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.535518 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.535519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:52.536391 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:52.536394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.536789 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.536790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:52.536913 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:52.537390 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:52.537398 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:52.538568 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:52.539939 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:52.539945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:52.539947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:52.540148 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:52.540314 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:52.540314 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:52.540315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:52.540419 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:52.540626 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:52.540659 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:52.540806 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 19:33:52.540896 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:33:52.541027 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4549,"thread_start_us":81,"threads_started":1}
I20260924 19:33:52.541707 13090 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40547
I20260924 19:33:52.541798 13090 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:52.541965 13090 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:52.542300 12606 ts_manager.cc:194] Registered new tserver with Master: 13c1ff28098b4244b97697bd84dde03c (127.0.98.3:33789)
I20260924 19:33:52.542656 12606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46485
I20260924 19:33:52.545181 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:33:52.545853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 12687
I20260924 19:33:52.550940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 12826
I20260924 19:33:52.555074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 12961
I20260924 19:33:52.559998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 12574
I20260924 19:33:52.564508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40547
--webserver_interface=127.0.98.62
--webserver_port=34895
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40547
--ranger_config_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58247/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58247/kms with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:52.642561 13093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:52.642782 13093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:52.642812 13093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:52.643970 13093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:33:52.644026 13093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:52.644048 13093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:52.644065 13093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:33:52.644080 13093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:33:52.645769 13093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58247/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40547
--ranger_config_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:52.646018 13093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:52.646231 13093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:52.650933 13098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:52.651062 13093 server_base.cc:1034] running on GCE node
W20260924 19:33:52.650946 13101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:52.650941 13099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:52.651381 13093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:52.651644 13093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:52.652777 13093 hybrid_clock.cc:648] HybridClock initialized: now 1790278432652765 us; error 34 us; skew 500 ppm
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:52.655162 13093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:33:52.655594 13093 webserver.cc:460] Webserver started at http://127.0.98.62:34895/ using document root <none> and password file <none>
I20260924 19:33:52.655779 13093 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:52.655833 13093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:52.703810 13093 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:33:52.704577 13109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.704764 13093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:33:52.704856 13093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "3b995882abfd4007a2340922fd424ce0"
format_stamp: "Formatted at 2026-09-24 19:33:50 on dist-test-slave-m8jb"
server_key: "ffbfb4e2cecba1dd4e53907f5e41d303"
server_key_iv: "68d06773a6a4d50e1f40a84732d09ca9"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:52.705225 13093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:52.722019 13093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:52.724079 13093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:52.724222 13093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:52.727943 13093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40547
I20260924 19:33:52.728003 13161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40547 every 8 connection(s)
I20260924 19:33:52.728338 13093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:33:52.729547 13162 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:33:52.729571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 13093
I20260924 19:33:52.729818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37595
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40547
--webserver_port=36057
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58247/kms with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:33:52.730285 13162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Bootstrap starting.
I20260924 19:33:52.737866 13162 log.cc:824] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:52.739500 13162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:33:52.739744 13162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Bootstrap complete.
I20260924 19:33:52.741942 13162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } }
I20260924 19:33:52.742170 13162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b995882abfd4007a2340922fd424ce0, State: Initialized, Role: FOLLOWER
I20260924 19:33:52.742326 13162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } }
I20260924 19:33:52.742465 13162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:33:52.742548 13162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:33:52.742607 13162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:33:52.743813 13162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } }
I20260924 19:33:52.743929 13162 leader_election.cc:302] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b995882abfd4007a2340922fd424ce0; no voters:
I20260924 19:33:52.744097 13162 leader_election.cc:288] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:33:52.744158 13166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:33:52.744293 13166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [term 2 LEADER]: Becoming Leader. State: Replica: 3b995882abfd4007a2340922fd424ce0, State: Running, Role: LEADER
I20260924 19:33:52.744369 13162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:33:52.744385 13166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } }
I20260924 19:33:52.744822 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3b995882abfd4007a2340922fd424ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } } }
I20260924 19:33:52.744992 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:52.744834 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b995882abfd4007a2340922fd424ce0. Latest consensus state: current_term: 2 leader_uuid: "3b995882abfd4007a2340922fd424ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b995882abfd4007a2340922fd424ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40547 } } }
I20260924 19:33:52.745128 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:52.745744 13162 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:33:52.747541 13176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:33:52.748376 13176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:33:52.748564 13176 catalog_manager.cc:1259] Loaded cluster ID: 5240428061e8451ea7f66e0d776b40d7
I20260924 19:33:52.748606 13176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:33:52.750471 13176 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:33:52.750684 13176 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Loaded TSK: 0
I20260924 19:33:52.750926 13176 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:33:52.813177 13164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:52.813380 13164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:52.813409 13164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:52.814644 13164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:52.814697 13164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:52.814734 13164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:33:52.816387 13164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37595
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36057
--tserver_master_addrs=127.0.98.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:52.816884 13164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:52.817201 13164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:52.823262 13199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:52.823305 13198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:52.823660 13164 server_base.cc:1034] running on GCE node
W20260924 19:33:52.824008 13201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:52.824261 13164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:52.824507 13164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:52.825670 13164 hybrid_clock.cc:648] HybridClock initialized: now 1790278432825592 us; error 91 us; skew 500 ppm
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:52.828066 13164 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:33:52.828507 13164 webserver.cc:460] Webserver started at http://127.0.98.1:36057/ using document root <none> and password file <none>
I20260924 19:33:52.828765 13164 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:52.828832 13164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:52.853789 13164 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:33:52.854815 13210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.855036 13164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:33:52.855144 13164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "e27d4ad82bd846c0a380aac4aaea0c3d"
format_stamp: "Formatted at 2026-09-24 19:33:51 on dist-test-slave-m8jb"
server_key: "9f9bc1cc1952874938d44577bd7093d4"
server_key_iv: "d434242cc09e8793fee0ed8e5fb4060a"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:52.855602 13164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:52.867168 13164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:52.869463 13164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:52.869614 13164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:52.869868 13164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:52.870227 13164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:52.870281 13164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.870313 13164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:52.870338 13164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:52.876469 13164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37595
I20260924 19:33:52.877034 13164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:33:52.879779 13323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37595 every 8 connection(s)
I20260924 19:33:52.885849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 13164
I20260924 19:33:52.886140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45223
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40547
--webserver_port=43333
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58247/kms with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:33:52 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:52.904270 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:52.881065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35201 (local address 127.0.98.62:40547)
0924 19:33:52.884800 (+ 3735us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:52.884813 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:52.885130 (+ 317us) server_negotiation.cc:407] Connection header received
0924 19:33:52.885961 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:52.885968 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:52.886030 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:52.886165 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:52.887013 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.888392 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.889933 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.890524 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.890603 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:52.898602 (+ 7999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:52.898609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:52.898609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:52.898625 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:52.900924 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:52.901538 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:52.901542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:52.901543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:52.901674 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:52.902518 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:52.902520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:52.902520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:52.902883 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:52.902960 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:52.903870 (+ 910us) server_negotiation.cc:299] Negotiation successful
0924 19:33:52.904018 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3660,"thread_start_us":93,"threads_started":1}
I20260924 19:33:52.905333 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:52.881329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40547 (local address 127.0.98.1:35201)
0924 19:33:52.884913 (+ 3584us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:52.884966 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:52.885631 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:52.886336 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:52.886340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:52.886593 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:52.886885 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:52.886892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.888532 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.888533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:52.889830 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:52.889833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:52.890736 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:52.890737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:52.890869 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:52.891646 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:52.891656 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:52.898378 (+ 6722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:52.901054 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:52.901065 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:52.901067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:52.901419 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:52.901769 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:52.901770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:52.901770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:52.901929 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:52.903000 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:52.903008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:52.903198 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 19:33:52.904888 (+ 1690us) client_negotiation.cc:240] Negotiation successful
0924 19:33:52.905087 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3541,"thread_start_us":75,"threads_started":1}
I20260924 19:33:52.907344 13324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40547
I20260924 19:33:52.907456 13324 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:52.907658 13324 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
W20260924 19:33:52.908766 13324 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40547 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:33:52.908847 13324 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:52.908917 13324 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:52.909376 13324 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:52.909448 13324 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
W20260924 19:33:52.909809 13324 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:33:52.994833 13328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:52.995185 13328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:52.995244 13328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:52.996526 13328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:52.996563 13328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:52.996590 13328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:33:52.998304 13328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43333
--tserver_master_addrs=127.0.98.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:52.998551 13328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:52.998865 13328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:53.005141 13336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:53.005427 13333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:53.005442 13328 server_base.cc:1034] running on GCE node
W20260924 19:33:53.005168 13334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:53.005918 13328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:53.006196 13328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:53.007342 13328 hybrid_clock.cc:648] HybridClock initialized: now 1790278433007308 us; error 48 us; skew 500 ppm
Sep 24 19:33:53 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:53.010556 13328 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:33:53.011101 13328 webserver.cc:460] Webserver started at http://127.0.98.2:43333/ using document root <none> and password file <none>
I20260924 19:33:53.011354 13328 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:53.011489 13328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:33:53 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:53.034894 13328 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.001s
I20260924 19:33:53.035746 13344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:53.035928 13328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:33:53.036006 13328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "66d9d85d573347c4b17243286e9e9427"
format_stamp: "Formatted at 2026-09-24 19:33:52 on dist-test-slave-m8jb"
server_key: "b5a3ae5af032f37f100e891362209720"
server_key_iv: "7bd4dbee60aee7e664d693fa03c466c6"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:53.036480 13328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:53.075672 13328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:53.077642 13328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:53.077777 13328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:53.077999 13328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:53.078347 13328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:53.078383 13328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:53.078413 13328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:53.078449 13328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:53.084446 13328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45223
I20260924 19:33:53.085564 13328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:33:53.086926 13457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45223 every 8 connection(s)
I20260924 19:33:53.091523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 13328
I20260924 19:33:53.091809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33789
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40547
--webserver_port=38045
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58247/kms with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:33:53 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:53.100684 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:53.088030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41731 (local address 127.0.98.62:40547)
0924 19:33:53.088074 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:53.088076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:53.088321 (+ 245us) server_negotiation.cc:407] Connection header received
0924 19:33:53.089188 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:53.089189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:53.089205 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:53.089234 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:53.090089 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.090833 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.094440 (+ 3607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.094739 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.094792 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:53.097343 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:53.097344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:53.097344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:53.097353 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:53.098942 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:53.099449 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:53.099450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:53.099450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:53.099516 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:53.099809 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:53.099809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:53.099810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:53.099965 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:53.100022 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:53.100447 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 19:33:53.100539 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:33:53.101451 13460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:53.087740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40547 (local address 127.0.98.2:41731)
0924 19:33:53.088183 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:53.088226 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:53.089063 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:53.089429 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:53.089433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:53.089681 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:53.089958 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:53.089965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.090927 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.090928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:53.094321 (+ 3393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:53.094324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.094901 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.094902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:53.095048 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:53.095581 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:53.095591 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:53.097208 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:53.099033 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:53.099040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:53.099043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:53.099357 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:53.099595 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:53.099595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:53.099596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:53.099728 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:53.100057 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:53.100063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:53.100291 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 19:33:53.101028 (+ 737us) client_negotiation.cc:240] Negotiation successful
0924 19:33:53.101201 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":88,"threads_started":1}
I20260924 19:33:53.102079 13458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40547
I20260924 19:33:53.102176 13458 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:53.102356 13458 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
W20260924 19:33:53.102962 13458 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40547 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:33:53.103030 13458 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:53.103091 13458 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:53.103438 13458 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:53.103506 13458 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
W20260924 19:33:53.103814 13458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:33:53.208182 13461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:53.208461 13461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:53.208503 13461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:53.210382 13461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:53.210446 13461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:53.210487 13461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:33:53.212806 13461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38045
--tserver_master_addrs=127.0.98.62:40547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:53.213195 13461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:53.213481 13461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:53.219664 13466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:53.219681 13469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:53.220055 13467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:53.220752 13461 server_base.cc:1034] running on GCE node
I20260924 19:33:53.220969 13461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:53.221253 13461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:53.222432 13461 hybrid_clock.cc:648] HybridClock initialized: now 1790278433222415 us; error 35 us; skew 500 ppm
Sep 24 19:33:53 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:53.225715 13461 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:33:53.226274 13461 webserver.cc:460] Webserver started at http://127.0.98.3:38045/ using document root <none> and password file <none>
I20260924 19:33:53.226585 13461 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:53.226719 13461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:33:53 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:53.264492 13461 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.001s sys 0.000s
I20260924 19:33:53.265411 13478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:53.265609 13461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:33:53.265681 13461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "13c1ff28098b4244b97697bd84dde03c"
format_stamp: "Formatted at 2026-09-24 19:33:52 on dist-test-slave-m8jb"
server_key: "3c8538dad55f86d8148d15033c46ace6"
server_key_iv: "b24b289d845d2b2827fa3f7cd179af15"
server_key_version: "kuduclusterkey@0"
I20260924 19:33:53.266132 13461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:53.304600 13461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:53.306517 13461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:53.306651 13461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:53.306914 13461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:33:53.307261 13461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:33:53.307330 13461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:53.307433 13461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:33:53.307468 13461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:53.315253 13461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33789
I20260924 19:33:53.315771 13461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:33:53.319484 13591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33789 every 8 connection(s)
I20260924 19:33:53.319895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 13461
Sep 24 19:33:53 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:53.326359 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:53.316865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56863 (local address 127.0.98.62:40547)
0924 19:33:53.316913 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:53.316915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:53.317152 (+ 237us) server_negotiation.cc:407] Connection header received
0924 19:33:53.317799 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:53.317801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:53.317814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:53.317843 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:53.318642 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.319383 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.320855 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.321064 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.321119 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:53.323387 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:53.323389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:53.323389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:53.323396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:53.324785 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:53.325234 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:53.325235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:53.325235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:53.325290 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:53.325585 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:53.325586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:53.325586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:53.325736 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:53.325811 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:53.326172 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 19:33:53.326258 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:33:53.326581 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:53.316881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40547 (local address 127.0.98.3:56863)
0924 19:33:53.317022 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:53.317065 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:53.317664 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:53.318029 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:53.318033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:53.318274 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:53.318542 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:53.318548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.319489 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.319490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:53.320787 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:53.320790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.321145 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.321146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:53.321269 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:53.321745 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:53.321755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:53.323296 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:53.324858 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:53.324865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:53.324867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:53.325145 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:53.325342 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:53.325342 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:53.325343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:53.325480 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:53.325816 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:53.325823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:53.326009 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 19:33:53.326136 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 19:33:53.326289 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
I20260924 19:33:53.327167 13592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40547
I20260924 19:33:53.327281 13592 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:53.327481 13592 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
W20260924 19:33:53.328194 13592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40547 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:33:53.328267 13592 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:53.328332 13592 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:53.328950 13592 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:53.329030 13592 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:53.329077 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:53.320343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.0.98.62:40547)
0924 19:33:53.320452 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:53.320453 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:53.320537 (+ 84us) server_negotiation.cc:407] Connection header received
0924 19:33:53.320614 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:53.320615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:53.320644 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:53.320695 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:53.321714 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.322427 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.324128 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:53.325069 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:53.325131 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:53.326398 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:53.326399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:53.326399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:53.326406 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:53.327873 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:53.328202 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:53.328203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:53.328203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:53.328268 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:53.328468 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:53.328469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:53.328469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:53.328591 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:53.328767 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:53.328833 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:33:53.328910 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
W20260924 19:33:53.329393 13592 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 19:33:53 dist-test-slave-m8jb krb5kdc[12130](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:53.909981 13324 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:53.910131 13324 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:53.910516 13125 ts_manager.cc:194] Registered new tserver with Master: e27d4ad82bd846c0a380aac4aaea0c3d (127.0.98.1:37595)
I20260924 19:33:53.911202 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35201
I20260924 19:33:54.103989 13458 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:54.104153 13458 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:54.104478 13125 ts_manager.cc:194] Registered new tserver with Master: 66d9d85d573347c4b17243286e9e9427 (127.0.98.2:45223)
I20260924 19:33:54.105046 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41731
Sep 24 19:33:54 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:54.329563 13592 heartbeater.cc:464] Registering TS with master...
I20260924 19:33:54.329711 13592 heartbeater.cc:511] Master 127.0.98.62:40547 requested a full tablet report, sending...
I20260924 19:33:54.330003 13125 ts_manager.cc:194] Registered new tserver with Master: 13c1ff28098b4244b97697bd84dde03c (127.0.98.3:33789)
I20260924 19:33:54.330526 13125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56863
I20260924 19:33:54.337234 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:33:54.345916 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.338167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.0.98.62:40547)
0924 19:33:54.338287 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:54.338289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:54.338353 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:33:54.338439 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:54.338439 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:54.338451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:54.338518 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:54.339171 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.339916 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.341044 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.341384 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.341446 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.342883 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:54.342885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:54.342885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:54.342892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:54.344564 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:54.344989 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:54.344990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:54.344991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:54.345051 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:54.345272 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:54.345273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:54.345273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:54.345400 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:54.345489 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:54.345619 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:33:54.345700 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260924 19:33:54.347889 13125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:33:54.350101 13125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:33:54.358860 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.353137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33789 (local address 127.0.0.1:34452)
0924 19:33:54.353989 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:54.354029 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:54.354098 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:54.354366 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:54.354369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:54.354372 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:54.354462 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.354465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.356057 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.356058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.357205 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.357208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.358520 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.358521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.358610 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.358626 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 19:33:54.358708 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:33:54.358769 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":86,"threads_started":1}
I20260924 19:33:54.359413 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.353548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.0.98.3:33789)
0924 19:33:54.353756 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:54.353761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:54.354093 (+ 332us) server_negotiation.cc:407] Connection header received
0924 19:33:54.354183 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:54.354186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:54.354204 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:54.354274 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:54.354538 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.355980 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.357298 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.358342 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.358410 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.358486 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:54.359177 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 19:33:54.359224 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
I20260924 19:33:54.360087 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.353400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46712 (local address 127.0.98.1:37595)
0924 19:33:54.353891 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:54.353895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:54.354285 (+ 390us) server_negotiation.cc:407] Connection header received
0924 19:33:54.354320 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:54.354323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:54.354337 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:54.354393 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:54.354570 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.355643 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.357081 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.357787 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.357855 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.357930 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:54.359621 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0924 19:33:54.359767 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":60,"threads_started":1}
I20260924 19:33:54.360375 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.353169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37595 (local address 127.0.0.1:46712)
0924 19:33:54.354189 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:54.354197 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:54.354239 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:54.354462 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:54.354463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:54.354465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:54.354506 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.354507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.355727 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.355728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.356991 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.356994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.359328 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.359329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.359393 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.359397 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:33:54.359451 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:33:54.359492 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":71,"threads_started":1}
I20260924 19:33:54.360502 13526 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0 (DEFAULT_TABLE table=test-table [id=1fcdd6c8b8784a1abbf0b810e0c24e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:33:54.360970 13526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:54.361150 13258 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0 (DEFAULT_TABLE table=test-table [id=1fcdd6c8b8784a1abbf0b810e0c24e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:33:54.361436 13258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:54.363857 13625 tablet_bootstrap.cc:492] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d: Bootstrap starting.
I20260924 19:33:54.364533 13625 tablet_bootstrap.cc:654] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:54.364913 13625 log.cc:824] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:54.365381 13626 tablet_bootstrap.cc:492] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: Bootstrap starting.
I20260924 19:33:54.365824 13625 tablet_bootstrap.cc:492] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d: No bootstrap required, opened a new log
I20260924 19:33:54.365891 13625 ts_tablet_manager.cc:1397] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:33:54.367885 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.353399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45223 (local address 127.0.0.1:57684)
0924 19:33:54.359670 (+ 6271us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:54.359681 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:54.359724 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:54.364291 (+ 4567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:54.364293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:54.364295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:54.364333 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.364334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.365542 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.365543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.366590 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.366593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.367646 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.367647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.367722 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.367725 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:33:54.367770 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:33:54.367813 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6257,"thread_start_us":69,"threads_started":1}
I20260924 19:33:54.368105 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.353485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.0.98.2:45223)
0924 19:33:54.360733 (+ 7248us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:54.360738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:54.360749 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:33:54.364091 (+ 3342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:54.364094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:54.364115 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:54.364202 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:54.364405 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.365456 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.366772 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.367476 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.367544 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.367614 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:54.367985 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 19:33:54.368033 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7200,"thread_start_us":43,"threads_started":1}
I20260924 19:33:54.368341 13625 raft_consensus.cc:348] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.368492 13625 raft_consensus.cc:374] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:54.368520 13625 raft_consensus.cc:729] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27d4ad82bd846c0a380aac4aaea0c3d, State: Initialized, Role: FOLLOWER
I20260924 19:33:54.368651 13625 consensus_queue.cc:260] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.368743 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0 (DEFAULT_TABLE table=test-table [id=1fcdd6c8b8784a1abbf0b810e0c24e3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:33:54.369059 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:54.369079 13625 ts_tablet_manager.cc:1428] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:33:54.369211 13324 heartbeater.cc:503] Master 127.0.98.62:40547 was elected leader, sending a full tablet report...
I20260924 19:33:54.369763 13626 tablet_bootstrap.cc:654] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:54.370092 13626 log.cc:824] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:54.371918 13626 tablet_bootstrap.cc:492] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: No bootstrap required, opened a new log
I20260924 19:33:54.371976 13626 ts_tablet_manager.cc:1397] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 19:33:54.374423 13626 raft_consensus.cc:348] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.374588 13626 raft_consensus.cc:374] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:54.374620 13626 raft_consensus.cc:729] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c1ff28098b4244b97697bd84dde03c, State: Initialized, Role: FOLLOWER
I20260924 19:33:54.374790 13626 consensus_queue.cc:260] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.375032 13626 ts_tablet_manager.cc:1428] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:33:54.375180 13592 heartbeater.cc:503] Master 127.0.98.62:40547 was elected leader, sending a full tablet report...
I20260924 19:33:54.375257 13630 tablet_bootstrap.cc:492] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427: Bootstrap starting.
I20260924 19:33:54.375902 13630 tablet_bootstrap.cc:654] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:54.376190 13630 log.cc:824] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:54.377027 13630 tablet_bootstrap.cc:492] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427: No bootstrap required, opened a new log
I20260924 19:33:54.377092 13630 ts_tablet_manager.cc:1397] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:33:54.379623 13630 raft_consensus.cc:348] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.379781 13630 raft_consensus.cc:374] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:54.379810 13630 raft_consensus.cc:729] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d9d85d573347c4b17243286e9e9427, State: Initialized, Role: FOLLOWER
I20260924 19:33:54.379938 13630 consensus_queue.cc:260] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.380173 13630 ts_tablet_manager.cc:1428] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:33:54.380203 13458 heartbeater.cc:503] Master 127.0.98.62:40547 was elected leader, sending a full tablet report...
W20260924 19:33:54.381147 13325 tablet.cc:2369] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:33:54 dist-test-slave-m8jb krb5kdc[12130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 19:33:54.481269 13628 raft_consensus.cc:482] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:33:54.481424 13628 raft_consensus.cc:504] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.481844 13628 leader_election.cc:288] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13c1ff28098b4244b97697bd84dde03c (127.0.98.3:33789), 66d9d85d573347c4b17243286e9e9427 (127.0.98.2:45223)
I20260924 19:33:54.486565 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.482101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45223 (local address 127.0.98.1:52173)
0924 19:33:54.482984 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:54.482993 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:54.483061 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:54.483222 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:54.483223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:54.483224 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:54.483276 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.483277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.484202 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.484202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.485508 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.485511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.486284 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.486284 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.486358 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.486362 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:33:54.486414 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:33:54.486455 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":123,"threads_started":1}
I20260924 19:33:54.486567 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.482489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52173 (local address 127.0.98.2:45223)
0924 19:33:54.482553 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:54.482572 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:54.483037 (+ 465us) server_negotiation.cc:407] Connection header received
0924 19:33:54.483129 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:54.483130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:54.483141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:54.483169 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:54.483340 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.484123 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.485592 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.486175 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.486213 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.486223 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:54.486442 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 19:33:54.486498 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:33:54.486943 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.482193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42999 (local address 127.0.98.3:33789)
0924 19:33:54.482237 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:54.482239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:54.482860 (+ 621us) server_negotiation.cc:407] Connection header received
0924 19:33:54.482975 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:54.482976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:54.482991 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:54.483022 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:33:54.483273 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.484009 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.485531 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.486100 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.486145 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.486157 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:54.486832 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 19:33:54.486882 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:33:54.487231 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3b9936ecf54c8ca77d8fccaeab78a0" candidate_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d9d85d573347c4b17243286e9e9427" is_pre_election: true
I20260924 19:33:54.487381 13412 raft_consensus.cc:2454] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e27d4ad82bd846c0a380aac4aaea0c3d in term 0.
I20260924 19:33:54.487496 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.482000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33789 (local address 127.0.98.1:42999)
0924 19:33:54.482797 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:54.482809 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:54.482893 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:54.483094 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:54.483095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:54.483097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:33:54.483199 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.483201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.484106 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.484106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.485427 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:54.485430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.486683 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.486684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:54.486759 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.486762 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:33:54.487368 (+ 606us) client_negotiation.cc:240] Negotiation successful
0924 19:33:54.487415 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":22,"thread_start_us":90,"threads_started":1}
I20260924 19:33:54.487624 13213 leader_election.cc:302] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d9d85d573347c4b17243286e9e9427, e27d4ad82bd846c0a380aac4aaea0c3d; no voters:
I20260924 19:33:54.487900 13628 raft_consensus.cc:2781] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:33:54.487955 13628 raft_consensus.cc:482] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:33:54.487978 13628 raft_consensus.cc:3037] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:54.488045 13528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3b9936ecf54c8ca77d8fccaeab78a0" candidate_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c1ff28098b4244b97697bd84dde03c" is_pre_election: true
I20260924 19:33:54.488193 13528 raft_consensus.cc:2454] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e27d4ad82bd846c0a380aac4aaea0c3d in term 0.
I20260924 19:33:54.488895 13628 raft_consensus.cc:504] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.489017 13628 leader_election.cc:288] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [CANDIDATE]: Term 1 election: Requested vote from peers 13c1ff28098b4244b97697bd84dde03c (127.0.98.3:33789), 66d9d85d573347c4b17243286e9e9427 (127.0.98.2:45223)
I20260924 19:33:54.489151 13528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3b9936ecf54c8ca77d8fccaeab78a0" candidate_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13c1ff28098b4244b97697bd84dde03c"
I20260924 19:33:54.489200 13412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3b9936ecf54c8ca77d8fccaeab78a0" candidate_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d9d85d573347c4b17243286e9e9427"
I20260924 19:33:54.489243 13528 raft_consensus.cc:3037] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:54.489265 13412 raft_consensus.cc:3037] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:54.490118 13412 raft_consensus.cc:2454] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e27d4ad82bd846c0a380aac4aaea0c3d in term 1.
I20260924 19:33:54.490111 13528 raft_consensus.cc:2454] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e27d4ad82bd846c0a380aac4aaea0c3d in term 1.
I20260924 19:33:54.490294 13213 leader_election.cc:302] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d9d85d573347c4b17243286e9e9427, e27d4ad82bd846c0a380aac4aaea0c3d; no voters:
I20260924 19:33:54.490399 13628 raft_consensus.cc:2781] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:33:54.490536 13628 raft_consensus.cc:686] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [term 1 LEADER]: Becoming Leader. State: Replica: e27d4ad82bd846c0a380aac4aaea0c3d, State: Running, Role: LEADER
I20260924 19:33:54.490630 13628 consensus_queue.cc:237] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } }
I20260924 19:33:54.491466 13125 catalog_manager.cc:5495] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d reported cstate change: term changed from 0 to 1, leader changed from <none> to e27d4ad82bd846c0a380aac4aaea0c3d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e27d4ad82bd846c0a380aac4aaea0c3d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 } health_report { overall_health: UNKNOWN } } }
I20260924 19:33:54.512941 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.508327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.0.98.1:37595)
0924 19:33:54.508370 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:54.508372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:54.509293 (+ 921us) server_negotiation.cc:407] Connection header received
0924 19:33:54.509346 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:54.509347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:54.509376 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:54.509442 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:33:54.509865 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.510583 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.511526 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:54.511795 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:54.511857 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:54.512413 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:33:54.512556 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:33:54.512624 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:54.512831 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 19:33:54.512855 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:33:54.523380 13412 raft_consensus.cc:1261] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 1 FOLLOWER]: Refusing update from remote peer e27d4ad82bd846c0a380aac4aaea0c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:33:54.523521 13528 raft_consensus.cc:1261] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 1 FOLLOWER]: Refusing update from remote peer e27d4ad82bd846c0a380aac4aaea0c3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:33:54.523813 13628 consensus_queue.cc:1035] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "13c1ff28098b4244b97697bd84dde03c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:33:54.523999 13628 consensus_queue.cc:1035] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P e27d4ad82bd846c0a380aac4aaea0c3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "66d9d85d573347c4b17243286e9e9427" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:33:54.531038 13641 mvcc.cc:205] Tried to move back new op lower bound from 7332980467804672000 to 7332980467674247168. Current Snapshot: MvccSnapshot[applied={T|T < 7332980467804672000}]
I20260924 19:33:54.531698 13642 mvcc.cc:205] Tried to move back new op lower bound from 7332980467804672000 to 7332980467674247168. Current Snapshot: MvccSnapshot[applied={T|T < 7332980467804672000}]
I20260924 19:33:54.534569 13125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:33:54.534669 13125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:33:54.535413 13125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b995882abfd4007a2340922fd424ce0: Sending DeleteTablet for 3 replicas of tablet 1e3b9936ecf54c8ca77d8fccaeab78a0
I20260924 19:33:54.535848 13526 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:33:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34452
I20260924 19:33:54.535910 13258 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:33:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46712
I20260924 19:33:54.536134 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:33:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57684
I20260924 19:33:54.536198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 13164
I20260924 19:33:54.540047 13651 tablet_replica.cc:330] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427: stopping tablet replica
I20260924 19:33:54.540212 13651 raft_consensus.cc:2229] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:33:54.540333 13651 raft_consensus.cc:2258] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 66d9d85d573347c4b17243286e9e9427 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:33:54.544085 13111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37595 (error 108)
I20260924 19:33:54.544158 13649 tablet_replica.cc:330] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: stopping tablet replica
I20260924 19:33:54.544258 13649 raft_consensus.cc:2229] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:33:54.544377 13649 raft_consensus.cc:2258] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:33:54.544559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 13328
I20260924 19:33:54.544579 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:54.544462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37595 (local address 127.0.0.1:46736)
0924 19:33:54.544507 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:54.544545 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:33:54.545450 13649 ts_tablet_manager.cc:1905] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 19:33:54.545987 13111 catalog_manager.cc:4584] TS e27d4ad82bd846c0a380aac4aaea0c3d (127.0.98.1:37595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e3b9936ecf54c8ca77d8fccaeab78a0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37595: connect: Connection refused (error 111)
I20260924 19:33:54.547003 13649 ts_tablet_manager.cc:1918] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:33:54.547122 13649 log.cc:1196] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/1e3b9936ecf54c8ca77d8fccaeab78a0
I20260924 19:33:54.547446 13649 ts_tablet_manager.cc:1939] T 1e3b9936ecf54c8ca77d8fccaeab78a0 P 13c1ff28098b4244b97697bd84dde03c: Deleting consensus metadata
I20260924 19:33:54.547946 13113 catalog_manager.cc:4841] TS 13c1ff28098b4244b97697bd84dde03c (127.0.98.3:33789): tablet 1e3b9936ecf54c8ca77d8fccaeab78a0 (table test-table [id=1fcdd6c8b8784a1abbf0b810e0c24e3d]) successfully deleted
W20260924 19:33:54.552286 13622 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:33:54.549563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45223 (local address 127.0.0.1:57698)
0924 19:33:54.549606 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:54.549617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:54.549653 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:54.552234 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45223: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 19:33:54.553321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 13461
I20260924 19:33:54.564857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 13093
I20260924 19:33:54.575732 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:33:54.909929 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:33:54.910017 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:33:55.244323 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:33:55.244 UTC [12160] LOG: received smart shutdown request
2026-09-24 19:33:55.248 UTC [12160] LOG: background worker "logical replication launcher" (PID 12169) exited with exit code 1
2026-09-24 19:33:55.248 UTC [12164] LOG: shutting down
2026-09-24 19:33:55.409 UTC [12160] LOG: database system is shut down
2026-09-24T19:33:55Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45752 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:33:55 dist-test-slave-m8jb krb5kdc[13661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:33:55 dist-test-slave-m8jb krb5kdc[13661](info): set up 2 sockets
Sep 24 19:33:55 dist-test-slave-m8jb krb5kdc[13661](info): commencing operation
krb5kdc: starting...
W20260924 19:33:56.444299 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:33:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:33:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:33:56.555925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43823
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43823,127.0.98.61:32793,127.0.98.60:38405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:56.634656 13674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:56.634865 13674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:56.634886 13674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:56.636016 13674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:33:56.636059 13674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:56.636072 13674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:56.636083 13674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:33:56.636094 13674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:33:56.637594 13674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43823,127.0.98.61:32793,127.0.98.60:38405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:56.637791 13674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:56.637970 13674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:56.642717 13679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:56.642727 13682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:56.642728 13680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:56.642938 13674 server_base.cc:1034] running on GCE node
I20260924 19:33:56.643229 13674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:56.643507 13674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:56.644665 13674 hybrid_clock.cc:648] HybridClock initialized: now 1790278436644642 us; error 31 us; skew 500 ppm
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:56.655747 13674 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:33:56.656212 13674 webserver.cc:460] Webserver started at http://127.0.98.62:44585/ using document root <none> and password file <none>
I20260924 19:33:56.656410 13674 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:56.656457 13674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:56.656584 13674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:33:56.657888 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "bbd8250f455940e8a900db256ba935b9"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "535c2dadfee8ba60562f5d17d98119b7"
server_key_iv: "73a1156c5be04ce77d0a8865203d0437"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.658228 13674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbd8250f455940e8a900db256ba935b9"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "535c2dadfee8ba60562f5d17d98119b7"
server_key_iv: "73a1156c5be04ce77d0a8865203d0437"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.659479 13674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:33:56.660171 13689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:56.660307 13674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:33:56.660392 13674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "bbd8250f455940e8a900db256ba935b9"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "535c2dadfee8ba60562f5d17d98119b7"
server_key_iv: "73a1156c5be04ce77d0a8865203d0437"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.660459 13674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:56.671100 13674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:56.672937 13674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:56.673064 13674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:56.676666 13674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43823
I20260924 19:33:56.676703 13741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43823 every 8 connection(s)
I20260924 19:33:56.677045 13674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:33:56.677824 13742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:56.679345 13742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:33:56.680305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 13674
I20260924 19:33:56.680430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:33:56.680603 392 external_mini_cluster.cc:1428] Setting key 79760787d4c2904a7c05773df3ab339d
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 19:33:56.688618 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.679971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43823 (local address 127.0.0.1:34922)
0924 19:33:56.680161 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.680199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:56.680735 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:56.681027 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:56.681030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:56.681265 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:56.681443 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.681448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.682560 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.682561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.683248 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.683250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.683750 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.683751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.683820 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.684371 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:56.684379 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:56.685722 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:56.687298 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.687304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.687306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.687485 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.687682 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.687682 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.687683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.687759 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.688105 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:56.688111 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:56.688175 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 19:33:56.688294 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 19:33:56.688486 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":70,"thread_start_us":81,"threads_started":1}
I20260924 19:33:56.688619 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.680005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.98.62:43823)
0924 19:33:56.680150 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.680160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.680290 (+ 130us) server_negotiation.cc:407] Connection header received
0924 19:33:56.680896 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.680898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.680920 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.681025 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.681526 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.682468 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.683329 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.683677 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.683772 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.685796 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.685803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.685804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.685810 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.687233 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.687552 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.687555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.687555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.687626 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.687809 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.687810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.687810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.688035 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.688112 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.688330 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.688424 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260924 19:33:56.689353 13742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 19:33:56.689663 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.689548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38002)
0924 19:33:56.689592 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.689640 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:33:56.689817 13690 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
I20260924 19:33:56.690238 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.690142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38010)
0924 19:33:56.690167 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.690223 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 19:33:56.690330 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32793: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 19:33:56.704037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32793
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43823,127.0.98.61:32793,127.0.98.60:38405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:33:56.739601 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } attempt: 1
I20260924 19:33:56.740082 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.739963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38024)
0924 19:33:56.740019 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.740051 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:33:56.740394 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.740297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38040)
0924 19:33:56.740331 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.740354 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 19:33:56.740482 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32793: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
W20260924 19:33:56.783317 13749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:56.783530 13749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:56.783558 13749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:56.784708 13749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:33:56.784745 13749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:56.784763 13749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:56.784778 13749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:33:56.784792 13749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:33:56.786260 13749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43823,127.0.98.61:32793,127.0.98.60:38405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:56.786500 13749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:56.786692 13749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:56.791200 13754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:56.791332 13755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:56.791347 13757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:56.791416 13749 server_base.cc:1034] running on GCE node
I20260924 19:33:56.791621 13749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:56.791888 13749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:56.793030 13749 hybrid_clock.cc:648] HybridClock initialized: now 1790278436793011 us; error 36 us; skew 500 ppm
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:56.795444 13749 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 19:33:56.795866 13749 webserver.cc:460] Webserver started at http://127.0.98.61:43295/ using document root <none> and password file <none>
I20260924 19:33:56.796056 13749 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:56.796096 13749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:56.796218 13749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:33:56.797291 13749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/data/instance:
uuid: "87be5ab1df594f7ca292f732ff390836"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "868babdd48ed2cf42fd6ca33bad88666"
server_key_iv: "3cfdcb3f8028f30238a0c3e0d7ddb26c"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.797609 13749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/wal/instance:
uuid: "87be5ab1df594f7ca292f732ff390836"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "868babdd48ed2cf42fd6ca33bad88666"
server_key_iv: "3cfdcb3f8028f30238a0c3e0d7ddb26c"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.798873 13749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:33:56.799542 13764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:56.799695 13749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:33:56.799773 13749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/wal
uuid: "87be5ab1df594f7ca292f732ff390836"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "868babdd48ed2cf42fd6ca33bad88666"
server_key_iv: "3cfdcb3f8028f30238a0c3e0d7ddb26c"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.799831 13749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:56.808696 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } attempt: 2
I20260924 19:33:56.809128 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.808983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38042)
0924 19:33:56.809044 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.809089 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:33:56.809535 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.809455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38046)
0924 19:33:56.809494 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.809517 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:33:56.809675 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32793: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32793: connect: Connection refused (error 111)
I20260924 19:33:56.813457 13749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:56.815299 13749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:56.815431 13749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:56.819494 13749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32793
I20260924 19:33:56.819525 13816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32793 every 8 connection(s)
I20260924 19:33:56.819842 13749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:33:56.820824 13817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:56.822372 13817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:56.828524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 13749
I20260924 19:33:56.828651 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-1/wal/instance
I20260924 19:33:56.828853 392 external_mini_cluster.cc:1428] Setting key aca181f762c706de05fce01990f2ac4c
I20260924 19:33:56.831223 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.823042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.0.98.62:43823)
0924 19:33:56.823104 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.823106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.823334 (+ 228us) server_negotiation.cc:407] Connection header received
0924 19:33:56.823801 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.823802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.823814 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.823852 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.824441 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.824973 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.825928 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.826123 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.826171 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.828383 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.828384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.828384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.828392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.829911 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.830327 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.830328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.830329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.830389 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.830603 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.830604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.830605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.830727 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.830784 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.831041 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.831131 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:33:56.831403 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.823014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43823 (local address 127.0.0.1:34932)
0924 19:33:56.823215 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.823253 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:56.823710 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:56.823907 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:56.823910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:56.824176 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:56.824371 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.824376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.825039 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.825039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.825854 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.825857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.826224 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.826225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.826336 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.826865 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:56.826874 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:56.828275 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:56.830000 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.830006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.830008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.830239 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.830446 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.830447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.830447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.830548 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.830784 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:56.830791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:56.830922 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 19:33:56.831010 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:33:56.831164 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260924 19:33:56.831863 13817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 19:33:56.838632 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.832096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38052)
0924 19:33:56.832149 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.832162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:56.832206 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:56.832485 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:56.832486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:56.832626 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:56.832707 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.832709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.833525 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.833526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.834305 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.834307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.834754 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.834755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.834832 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.835187 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:56.835193 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:56.836368 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:56.837781 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.837782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.837782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.837927 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.838069 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.838070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.838070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.838111 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.838365 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:56.838366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:56.838407 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:33:56.838473 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 19:33:56.838557 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:33:56.838709 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.832142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.0.98.61:32793)
0924 19:33:56.832343 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.832348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.832366 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:33:56.832387 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.832389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.832402 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.832465 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.832781 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.833450 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.834385 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.834675 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.834746 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.836446 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.836453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.836453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.836459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.837712 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.837985 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.837987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.837987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.838030 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.838143 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.838144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.838144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.838280 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.838348 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.838508 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.838616 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 19:33:56.839285 13817 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:33:56.839613 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.839512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46442)
0924 19:33:56.839552 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.839594 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:33:56.839745 13768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
I20260924 19:33:56.840126 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.840042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46450)
0924 19:33:56.840081 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.840104 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 19:33:56.840257 13817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38405: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 19:33:56.853660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38405
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43823,127.0.98.61:32793,127.0.98.60:38405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:33:56.889482 13817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } attempt: 1
I20260924 19:33:56.889988 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.889860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46458)
0924 19:33:56.889917 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.889958 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:33:56.890319 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.890223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46474)
0924 19:33:56.890272 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.890296 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:33:56.890405 13817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38405: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
I20260924 19:33:56.900820 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } attempt: 3
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:33:56.907605 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.901178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38058 (local address 127.0.98.61:32793)
0924 19:33:56.901217 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.901218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.901228 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:33:56.901303 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.901304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.901312 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.901350 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.901684 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.902220 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.903240 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.903506 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.903557 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.905424 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.905426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.905426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.905432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.906628 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.906945 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.906946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.906947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.906996 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.907153 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.907154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.907155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.907255 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.907307 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.907440 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.907509 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:33:56.907606 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.901126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38058)
0924 19:33:56.901183 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.901194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:56.901239 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:56.901364 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:56.901366 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:56.901539 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:56.901609 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.901611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.902352 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.902353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.903151 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.903154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.903567 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.903568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.903660 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.904065 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:56.904071 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:56.905338 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:56.906701 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.906703 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.906703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.906886 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.907048 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.907049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.907049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.907105 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.907307 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:56.907307 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:56.907352 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:33:56.907412 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:33:56.907509 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:33:56.907948 13742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:33:56.908244 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.908156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46482)
0924 19:33:56.908193 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.908218 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:33:56.908569 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.908497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46496)
0924 19:33:56.908534 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.908555 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:33:56.908668 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38405: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
W20260924 19:33:56.933439 13824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:56.933653 13824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:56.933677 13824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:56.934760 13824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:33:56.934791 13824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:56.934803 13824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:56.934814 13824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:33:56.934827 13824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:33:56.936225 13824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43823,127.0.98.61:32793,127.0.98.60:38405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:56.936401 13824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:56.936566 13824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:33:56.939832 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } attempt: 1
I20260924 19:33:56.940315 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.940166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46502)
0924 19:33:56.940244 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.940282 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 19:33:56.940761 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.940592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46514)
0924 19:33:56.940696 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.940728 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260924 19:33:56.940896 13742 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38405: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
W20260924 19:33:56.940999 13830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:56.940973 13829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:56.941084 13832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:56.941295 13824 server_base.cc:1034] running on GCE node
I20260924 19:33:56.941529 13824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:56.941820 13824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:56.942999 13824 hybrid_clock.cc:648] HybridClock initialized: now 1790278436942973 us; error 36 us; skew 500 ppm
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:56.945457 13824 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 19:33:56.945896 13824 webserver.cc:460] Webserver started at http://127.0.98.60:40393/ using document root <none> and password file <none>
I20260924 19:33:56.946087 13824 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:56.946132 13824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:56.946252 13824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:33:56.947263 13824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/data/instance:
uuid: "a417318bb81e4faf82d4c06b42584786"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "c5a7b277caece48cf03f3256f15ede30"
server_key_iv: "212af7c30aa6e1d1b5b4872d84499ee7"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.947623 13824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/wal/instance:
uuid: "a417318bb81e4faf82d4c06b42584786"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "c5a7b277caece48cf03f3256f15ede30"
server_key_iv: "212af7c30aa6e1d1b5b4872d84499ee7"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.948818 13824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:33:56.949472 13839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:56.949615 13824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:33:56.949692 13824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/wal
uuid: "a417318bb81e4faf82d4c06b42584786"
format_stamp: "Formatted at 2026-09-24 19:33:56 on dist-test-slave-m8jb"
server_key: "c5a7b277caece48cf03f3256f15ede30"
server_key_iv: "212af7c30aa6e1d1b5b4872d84499ee7"
server_key_version: "encryptionkey@0"
I20260924 19:33:56.949751 13824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:56.958529 13817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } attempt: 2
I20260924 19:33:56.958881 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.958779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46530)
0924 19:33:56.958828 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.958855 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:33:56.959167 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.959092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46532)
0924 19:33:56.959126 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.959149 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 19:33:56.959244 13817 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38405: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38405: connect: Connection refused (error 111)
I20260924 19:33:56.965569 13824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:56.967358 13824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:56.967501 13824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:56.971593 13824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38405
I20260924 19:33:56.971625 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38405 every 8 connection(s)
I20260924 19:33:56.971964 13824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:33:56.972824 13892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:56.974352 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:33:56.977814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 13824
I20260924 19:33:56.977905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790278290121024-392-0/minicluster-data/master-2/wal/instance
I20260924 19:33:56.978060 392 external_mini_cluster.cc:1428] Setting key ef8d985de0c6cea6da15187cdb74f41a
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:33:56.983129 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.975067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34944 (local address 127.0.98.62:43823)
0924 19:33:56.975215 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.975218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.975354 (+ 136us) server_negotiation.cc:407] Connection header received
0924 19:33:56.975847 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.975848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.975860 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.975905 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.976524 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.977058 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.978050 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.978265 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.978318 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.980535 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.980536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.980537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.980543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.981906 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.982254 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.982255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.982256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.982306 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.982538 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.982539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.982539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.982642 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.982695 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.982958 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.983038 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:33:56.983278 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.974998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43823 (local address 127.0.0.1:34944)
0924 19:33:56.975210 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.975259 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:56.975752 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:56.975990 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:56.975994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:56.976243 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:56.976438 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.976442 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.977139 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.977140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.977966 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.977969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.978333 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.978334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.978464 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.979036 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:56.979044 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:56.980459 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:56.981973 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.981980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.981982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.982190 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.982379 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.982381 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.982381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.982486 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.982696 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:56.982703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:56.982833 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 19:33:56.982930 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:33:56.983068 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
I20260924 19:33:56.983819 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:33:56.986106 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.978363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46540 (local address 127.0.98.60:38405)
0924 19:33:56.978528 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.978532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.978611 (+ 79us) server_negotiation.cc:407] Connection header received
0924 19:33:56.978648 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.978651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.978672 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.978734 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.979357 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.980172 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.981098 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.981548 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.981622 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.983576 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.983581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.983581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.983588 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.985081 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.985417 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.985420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.985420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.985465 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.985621 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.985621 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.985622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.985740 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.985855 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.985936 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.986003 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:33:56.990147 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.984056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32793 (local address 127.0.0.1:38066)
0924 19:33:56.984094 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.984103 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:56.984137 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:56.984284 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:56.984285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:56.984405 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:56.984446 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.984447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.985077 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.985078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.985760 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.985761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.986303 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.986304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.986375 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.986695 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:56.986700 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:56.987857 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:56.989276 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.989277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.989277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.989458 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.989644 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.989645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.989645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.989696 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.989945 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:56.989945 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:56.989976 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:33:56.990018 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 19:33:56.990083 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 19:33:56.990218 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.984131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38066 (local address 127.0.98.61:32793)
0924 19:33:56.984193 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.984195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.984202 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:33:56.984226 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.984226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.984236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.984277 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.984508 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.985014 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.985840 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.986116 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.986184 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.987927 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.987928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.987929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.987936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.989208 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.989531 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.989532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.989533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.989590 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.989748 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.989750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.989750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.989886 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.989956 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.990047 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.990127 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:33:56.990636 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:56.994011 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.986949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34950 (local address 127.0.98.62:43823)
0924 19:33:56.986982 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.986984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.986993 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:33:56.987013 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.987014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.987022 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.987066 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.987491 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.988063 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.988962 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.989135 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.989184 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.991511 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.991513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.991513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.991520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.992961 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.993284 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.993285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.993285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.993334 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.993509 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.993509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.993510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.993618 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.993813 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.993849 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.993918 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:33:56.996850 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.990865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46546)
0924 19:33:56.990965 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:56.990975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:56.991022 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:56.991134 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:56.991135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:56.991264 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:56.991301 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.991302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.992120 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.992121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.992715 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:56.992718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.993158 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.993159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:56.993214 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.993533 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:56.993538 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:56.994693 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:56.995993 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.995994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.995994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.996168 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.996335 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:56.996336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:56.996336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:56.996383 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:56.996611 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:56.996612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:56.996672 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 19:33:56.996726 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:33:56.996788 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:33:56.996953 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.990927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.0.98.60:38405)
0924 19:33:56.990962 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.990964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.991008 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:33:56.991071 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.991071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.991080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.991130 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.991380 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.992042 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.992783 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.993097 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.993144 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.994752 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.994753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.994754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.994761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:56.995922 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.996220 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.996221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.996221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.996277 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:56.996431 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:56.996431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:56.996432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:56.996561 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:56.996618 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:56.996760 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 19:33:56.996856 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":17}
I20260924 19:33:56.998649 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786: Bootstrap starting.
Sep 24 19:33:56 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:33:56.999533 13892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:56.999831 13892 log.cc:824] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:57.000757 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786: No bootstrap required, opened a new log
I20260924 19:33:57.002004 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:56.994844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.0.98.61:32793)
0924 19:33:56.994880 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:56.994881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:56.994905 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:33:56.994924 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:56.994925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:56.994934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:56.994978 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:56.995542 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.995999 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.996885 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:56.997142 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:56.997201 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:56.999433 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:56.999434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:56.999434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:56.999442 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:57.001007 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.001288 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.001289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.001290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.001347 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.001476 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.001476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.001477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.001603 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:57.001789 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:57.001828 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 19:33:57.001914 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:33:57.002717 13892 raft_consensus.cc:348] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.002875 13892 raft_consensus.cc:374] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:57.002919 13892 raft_consensus.cc:729] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a417318bb81e4faf82d4c06b42584786, State: Initialized, Role: FOLLOWER
I20260924 19:33:57.003038 13892 consensus_queue.cc:260] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.003290 13898 sys_catalog.cc:455] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.003337 13892 sys_catalog.cc:564] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:33:57.003423 13898 sys_catalog.cc:458] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:33:57.003469 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 19:33:57.005023 13912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:33:57.005385 13912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:33:57.005903 13824 master_runner.cc:186] Error getting master registration for 127.0.98.62:43823: OK, instance_id { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" instance_seqno: 1790278436675647 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:33:57.008065 13824 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:33:57.012372 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:57.005170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.0.98.62:43823)
0924 19:33:57.005338 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:57.005340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:57.005347 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:33:57.005368 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:57.005369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:57.005377 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:57.005418 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:57.006072 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.006764 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.007737 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.008019 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.008063 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:57.009797 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:57.009798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:57.009798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:57.009806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:57.011431 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.011713 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.011714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.011715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.011768 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.011924 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.011925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.011925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.012026 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:57.012120 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:57.012199 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:33:57.012271 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260924 19:33:57.014922 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:57.005169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.98.60:38405)
0924 19:33:57.005207 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:57.005209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:57.005217 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:33:57.005235 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:57.005236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:57.005245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:57.005288 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:57.008949 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.009688 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.010629 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.010821 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.010872 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:57.012139 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:57.012140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:57.012140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:57.012147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:57.013649 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.013997 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.013998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.013998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.014053 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.014227 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.014228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.014228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.014398 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:57.014492 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:57.014743 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 19:33:57.014832 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:33:57.016026 13742 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } attempt: 2
I20260924 19:33:57.020337 13821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:57.005242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.98.61:32793)
0924 19:33:57.005460 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:57.005462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:57.012660 (+ 7198us) server_negotiation.cc:407] Connection header received
0924 19:33:57.012755 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:57.012756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:57.012767 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:57.012794 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:57.013518 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.014086 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.015147 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.015353 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.015414 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:57.016815 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:57.016817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:57.016817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:57.016825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:57.018876 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.019280 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.019281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.019282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.019354 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.019766 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.019768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.019768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.019900 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:57.020021 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:57.020167 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:33:57.020244 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
Sep 24 19:33:57 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:33:57.023118 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:57.016259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46564)
0924 19:33:57.016315 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:57.016325 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:57.016384 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:57.016572 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:57.016573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:57.016767 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:57.016812 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:57.016814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.017882 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.017884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:57.018873 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:57.018876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.019218 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.019219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:57.019271 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:57.019604 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:57.019609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:57.020927 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:57.022248 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:57.022249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:57.022250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:57.022406 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:57.022562 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:57.022562 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:57.022563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:57.022618 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:57.022817 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:57.022818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:57.022869 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 19:33:57.022927 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:33:57.023021 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:33:57.023115 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:57.016438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.0.98.60:38405)
0924 19:33:57.016484 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:57.016486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:57.016492 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:33:57.016510 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:57.016510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:57.016518 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:57.016560 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:57.016889 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.017560 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.018970 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.019159 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.019209 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:57.021014 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:57.021015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:57.021015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:57.021022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:57.022175 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.022463 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.022464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.022464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.022511 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.022665 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.022666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.022667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.022764 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:57.022818 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:57.022954 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 19:33:57.023022 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:33:57.024338 13742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: Bootstrap starting.
I20260924 19:33:57.025013 13742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:57.025290 13742 log.cc:824] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:57.026034 13742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: No bootstrap required, opened a new log
I20260924 19:33:57.027599 13742 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.027709 13742 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:57.027729 13742 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd8250f455940e8a900db256ba935b9, State: Initialized, Role: FOLLOWER
I20260924 19:33:57.027855 13742 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.028158 13742 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:33:57.028139 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.028306 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:33:57.029821 13930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:33:57.029933 13930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:33:57.030498 13674 master_runner.cc:186] Error getting master registration for 127.0.98.61:32793: OK, instance_id { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" instance_seqno: 1790278436818401 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:33:57.032851 13674 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:33:57.050396 13817 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } attempt: 3
Sep 24 19:33:57 dist-test-slave-m8jb krb5kdc[13661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:33:57.056936 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:57.050865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.0.98.60:38405)
0924 19:33:57.051056 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:57.051058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:57.051065 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:33:57.051083 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:57.051084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:57.051093 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:57.051447 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:57.051476 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.051937 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.052734 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.052924 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.052977 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:57.054722 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:57.054723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:57.054724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:57.054731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:57.055942 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.056232 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.056233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.056233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.056288 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:57.056446 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:57.056447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:57.056447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:57.056556 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:57.056611 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:57.056760 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 19:33:57.056835 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260924 19:33:57.056936 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:57.050735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38405 (local address 127.0.0.1:46568)
0924 19:33:57.050786 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 19:33:57.050799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:33:57.050835 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:33:57.051175 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:33:57.051177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:33:57.051342 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:33:57.051391 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:57.051393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.052038 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.052040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:57.052666 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:33:57.052669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:33:57.052978 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:33:57.052980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:33:57.053051 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:57.053487 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:33:57.053492 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:33:57.054648 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:33:57.056016 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:57.056017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:57.056018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:57.056176 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:57.056340 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:33:57.056341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:33:57.056341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:33:57.056399 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:33:57.056607 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:33:57.056608 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:33:57.056675 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 19:33:57.056735 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:33:57.056835 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:33:57.058509 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: Bootstrap starting.
I20260924 19:33:57.059121 13817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:57.059397 13817 log.cc:824] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:57.060511 13817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: No bootstrap required, opened a new log
I20260924 19:33:57.061913 13817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.062021 13817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:57.062053 13817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87be5ab1df594f7ca292f732ff390836, State: Initialized, Role: FOLLOWER
I20260924 19:33:57.062184 13817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.062456 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.062515 13817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:33:57.062608 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:33:57.064390 13944 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:33:57.064486 13944 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:33:57.066247 13770 tablet.cc:2369] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:33:57.066716 13749 master_runner.cc:263] No leader master found from master 87be5ab1df594f7ca292f732ff390836
I20260924 19:33:57.066756 13749 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:33:57.166723 13898 raft_consensus.cc:482] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:33:57.166855 13898 raft_consensus.cc:504] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.167079 13898 leader_election.cc:288] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbd8250f455940e8a900db256ba935b9 (127.0.98.62:43823), 87be5ab1df594f7ca292f732ff390836 (127.0.98.61:32793)
I20260924 19:33:57.167737 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a417318bb81e4faf82d4c06b42584786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87be5ab1df594f7ca292f732ff390836" is_pre_election: true
I20260924 19:33:57.167909 13791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a417318bb81e4faf82d4c06b42584786 in term 0.
I20260924 19:33:57.168246 13840 leader_election.cc:302] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87be5ab1df594f7ca292f732ff390836, a417318bb81e4faf82d4c06b42584786; no voters:
I20260924 19:33:57.168469 13898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:33:57.168529 13898 raft_consensus.cc:482] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:33:57.168548 13898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:57.169591 13898 raft_consensus.cc:504] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.169716 13898 leader_election.cc:288] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [CANDIDATE]: Term 1 election: Requested vote from peers bbd8250f455940e8a900db256ba935b9 (127.0.98.62:43823), 87be5ab1df594f7ca292f732ff390836 (127.0.98.61:32793)
I20260924 19:33:57.169974 13791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a417318bb81e4faf82d4c06b42584786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87be5ab1df594f7ca292f732ff390836"
I20260924 19:33:57.170046 13791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:57.170912 13791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a417318bb81e4faf82d4c06b42584786 in term 1.
I20260924 19:33:57.171123 13840 leader_election.cc:302] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87be5ab1df594f7ca292f732ff390836, a417318bb81e4faf82d4c06b42584786; no voters:
I20260924 19:33:57.171473 13715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a417318bb81e4faf82d4c06b42584786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd8250f455940e8a900db256ba935b9"
I20260924 19:33:57.171607 13715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:57.172461 13715 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a417318bb81e4faf82d4c06b42584786 in term 1.
I20260924 19:33:57.172614 13716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a417318bb81e4faf82d4c06b42584786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbd8250f455940e8a900db256ba935b9" is_pre_election: true
I20260924 19:33:57.172712 13716 raft_consensus.cc:2362] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a417318bb81e4faf82d4c06b42584786 in term 1. Re-sending same reply.
I20260924 19:33:57.173022 13898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:33:57.173163 13898 raft_consensus.cc:686] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [term 1 LEADER]: Becoming Leader. State: Replica: a417318bb81e4faf82d4c06b42584786, State: Running, Role: LEADER
I20260924 19:33:57.173241 13898 consensus_queue.cc:237] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } }
I20260924 19:33:57.173614 13898 sys_catalog.cc:455] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a417318bb81e4faf82d4c06b42584786. Latest consensus state: current_term: 1 leader_uuid: "a417318bb81e4faf82d4c06b42584786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.173707 13898 sys_catalog.cc:458] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: This master's current role is: LEADER
W20260924 19:33:57.173902 13695 tablet.cc:2369] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:33:57.174319 13948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:33:57.174633 13948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:33:57.177624 13791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [term 1 FOLLOWER]: Refusing update from remote peer a417318bb81e4faf82d4c06b42584786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:33:57.177850 13716 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [term 1 FOLLOWER]: Refusing update from remote peer a417318bb81e4faf82d4c06b42584786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:33:57.178198 13898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:33:57.178539 13898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:33:57.179751 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a417318bb81e4faf82d4c06b42584786. Latest consensus state: current_term: 1 leader_uuid: "a417318bb81e4faf82d4c06b42584786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.179836 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:33:57.180902 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a417318bb81e4faf82d4c06b42584786. Latest consensus state: current_term: 1 leader_uuid: "a417318bb81e4faf82d4c06b42584786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.180996 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:33:57.182276 13898 sys_catalog.cc:455] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a417318bb81e4faf82d4c06b42584786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.182372 13898 sys_catalog.cc:458] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:57.182754 13919 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a417318bb81e4faf82d4c06b42584786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.182830 13919 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:33:57.183055 13949 mvcc.cc:205] Tried to move back new op lower bound from 7332980478676856832 to 7332980478662152192. Current Snapshot: MvccSnapshot[applied={T|T < 7332980478676856832}]
I20260924 19:33:57.183568 13950 mvcc.cc:205] Tried to move back new op lower bound from 7332980478676856832 to 7332980478662152192. Current Snapshot: MvccSnapshot[applied={T|T < 7332980478676856832}]
I20260924 19:33:57.185143 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a417318bb81e4faf82d4c06b42584786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.185227 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786 [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:57.185684 13952 mvcc.cc:205] Tried to move back new op lower bound from 7332980478676856832 to 7332980478662152192. Current Snapshot: MvccSnapshot[applied={T|T < 7332980478676856832}]
I20260924 19:33:57.185760 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a417318bb81e4faf82d4c06b42584786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbd8250f455940e8a900db256ba935b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43823 } } peers { permanent_uuid: "87be5ab1df594f7ca292f732ff390836" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32793 } } peers { permanent_uuid: "a417318bb81e4faf82d4c06b42584786" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38405 } } }
I20260924 19:33:57.185819 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:33:57.186162 13948 catalog_manager.cc:1347] Generated new cluster ID: 45ed9178da744c2c89ca462a0de3e5e7
I20260924 19:33:57.186201 13948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:33:57.202143 13948 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:33:57.202729 13948 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:33:57.205804 13948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a417318bb81e4faf82d4c06b42584786: Generated new TSK 0
I20260924 19:33:57.206007 13948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:33:58.030525 13930 catalog_manager.cc:1259] Loaded cluster ID: 45ed9178da744c2c89ca462a0de3e5e7
I20260924 19:33:58.030612 13930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: loading cluster ID for follower catalog manager: success
I20260924 19:33:58.031473 13930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: acquiring CA information for follower catalog manager: success
I20260924 19:33:58.031787 13930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbd8250f455940e8a900db256ba935b9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:33:58.064991 13944 catalog_manager.cc:1259] Loaded cluster ID: 45ed9178da744c2c89ca462a0de3e5e7
I20260924 19:33:58.065073 13944 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: loading cluster ID for follower catalog manager: success
I20260924 19:33:58.065842 13944 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: acquiring CA information for follower catalog manager: success
I20260924 19:33:58.066114 13944 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87be5ab1df594f7ca292f732ff390836: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:33:58.261070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 13674
I20260924 19:33:58.266822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 13749
I20260924 19:33:58.271998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 13824
2026-09-24T19:33:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2775 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:33:58 dist-test-slave-m8jb krb5kdc[13963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:33:58 dist-test-slave-m8jb krb5kdc[13963](info): set up 2 sockets
Sep 24 19:33:58 dist-test-slave-m8jb krb5kdc[13963](info): commencing operation
krb5kdc: starting...
W20260924 19:33:59.228545 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:33:59 dist-test-slave-m8jb krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:33:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:33:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:33:59.339629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:33:59.417704 13976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:33:59.417919 13976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:33:59.417949 13976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:33:59.419083 13976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:33:59.419122 13976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:33:59.419140 13976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:33:59.419157 13976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:33:59.419173 13976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:33:59.420620 13976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:33:59.420866 13976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:33:59.421069 13976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:33:59.425473 13981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:59.425482 13984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:33:59.425523 13982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:33:59.425583 13976 server_base.cc:1034] running on GCE node
I20260924 19:33:59.425935 13976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:33:59.426206 13976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:33:59.427340 13976 hybrid_clock.cc:648] HybridClock initialized: now 1790278439427327 us; error 29 us; skew 500 ppm
Sep 24 19:33:59 dist-test-slave-m8jb krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:33:59.429772 13976 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:33:59.430192 13976 webserver.cc:460] Webserver started at http://127.0.98.62:44893/ using document root <none> and password file <none>
I20260924 19:33:59.430400 13976 fs_manager.cc:359] Metadata directory not provided
I20260924 19:33:59.430449 13976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:33:59.430567 13976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:33:59.431594 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7"
format_stamp: "Formatted at 2026-09-24 19:33:59 on dist-test-slave-m8jb"
server_key: "2b827069d028c1c22c19d9a6bfe43306"
server_key_iv: "f1cd2d871e5584a0d3dd63b5387a8845"
server_key_version: "encryptionkey@0"
I20260924 19:33:59.431934 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7"
format_stamp: "Formatted at 2026-09-24 19:33:59 on dist-test-slave-m8jb"
server_key: "2b827069d028c1c22c19d9a6bfe43306"
server_key_iv: "f1cd2d871e5584a0d3dd63b5387a8845"
server_key_version: "encryptionkey@0"
I20260924 19:33:59.433238 13976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:33:59.433904 13991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:33:59.434048 13976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:33:59.434126 13976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7"
format_stamp: "Formatted at 2026-09-24 19:33:59 on dist-test-slave-m8jb"
server_key: "2b827069d028c1c22c19d9a6bfe43306"
server_key_iv: "f1cd2d871e5584a0d3dd63b5387a8845"
server_key_version: "encryptionkey@0"
I20260924 19:33:59.434183 13976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:33:59.452263 13976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:33:59.454353 13976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:33:59.454489 13976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:33:59.458213 13976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33995
I20260924 19:33:59.458250 14043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33995 every 8 connection(s)
I20260924 19:33:59.458573 13976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:33:59.459374 14044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:33:59.461611 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7: Bootstrap starting.
I20260924 19:33:59.462137 14044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7: Neither blocks nor log segments found. Creating new log.
I20260924 19:33:59.462396 14044 log.cc:824] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:33:59.463004 14044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7: No bootstrap required, opened a new log
I20260924 19:33:59.464064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 13976
I20260924 19:33:59.464145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:33:59.464304 392 external_mini_cluster.cc:1428] Setting key 01a85a43fa02ebe80633f38c95ce192c
I20260924 19:33:59.464761 14044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33995 } }
I20260924 19:33:59.464937 14044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:33:59.464974 14044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee92ab6b3fd49fdbcf95145d20c10f7, State: Initialized, Role: FOLLOWER
I20260924 19:33:59.465122 14044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33995 } }
I20260924 19:33:59.465198 14044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:33:59.465232 14044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:33:59.465278 14044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:33:59.466221 14044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33995 } }
I20260924 19:33:59.466322 14044 leader_election.cc:302] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee92ab6b3fd49fdbcf95145d20c10f7; no voters:
I20260924 19:33:59.466483 14044 leader_election.cc:288] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:33:59.466511 14049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:33:59.466650 14049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee92ab6b3fd49fdbcf95145d20c10f7, State: Running, Role: LEADER
I20260924 19:33:59.466706 14044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:33:59.466782 14049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33995 } }
I20260924 19:33:59.467090 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ee92ab6b3fd49fdbcf95145d20c10f7. Latest consensus state: current_term: 1 leader_uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33995 } } }
I20260924 19:33:59.467301 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:59.467231 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee92ab6b3fd49fdbcf95145d20c10f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33995 } } }
I20260924 19:33:59.467530 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:33:59.467571 14058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:33:59.468044 14058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:33:59.469846 14058 catalog_manager.cc:1347] Generated new cluster ID: 404aa9a01d254900993d13b4d46d6475
I20260924 19:33:59.469894 14058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:33:59 dist-test-slave-m8jb krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:33:59.475061 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:33:59.464781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.0.98.62:33995)
0924 19:33:59.464962 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:33:59.464971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:33:59.465008 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:33:59.465121 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:33:59.465128 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:33:59.465184 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:33:59.465328 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:33:59.465890 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:59.467225 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:59.468255 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:33:59.468878 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:33:59.468989 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:33:59.471624 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:33:59.471628 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:33:59.471628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:33:59.471640 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:33:59.473518 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:59.473915 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:59.473918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:59.473919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:59.474030 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:33:59.474222 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:33:59.474222 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:33:59.474222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:33:59.474498 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:33:59.474635 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:33:59.474744 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 19:33:59.474872 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260924 19:33:59.475966 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:33:59.478776 14058 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:33:59.479480 14058 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:33:59.483206 14058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ee92ab6b3fd49fdbcf95145d20c10f7: Generated new TSK 0
I20260924 19:33:59.483361 14058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:33:59.529711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 13976
2026-09-24T19:33:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1255 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:33:59 dist-test-slave-m8jb krb5kdc[14071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:33:59 dist-test-slave-m8jb krb5kdc[14071](info): set up 2 sockets
Sep 24 19:33:59 dist-test-slave-m8jb krb5kdc[14071](info): commencing operation
krb5kdc: starting...
W20260924 19:34:00.476909 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:34:00 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:34:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:34:00.591427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:00.669924 14084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:00.670133 14084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:00.670163 14084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:00.671314 14084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:34:00.671360 14084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:00.671377 14084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:00.671393 14084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:34:00.671407 14084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:34:00.672904 14084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:00.673146 14084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:00.673357 14084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:00.678207 14090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:00.678216 14092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:00.678216 14089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:00.678305 14084 server_base.cc:1034] running on GCE node
I20260924 19:34:00.678639 14084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:00.678908 14084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:00.680047 14084 hybrid_clock.cc:648] HybridClock initialized: now 1790278440680036 us; error 32 us; skew 500 ppm
Sep 24 19:34:00 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:00.682446 14084 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:34:00.682869 14084 webserver.cc:460] Webserver started at http://127.0.98.62:40277/ using document root <none> and password file <none>
I20260924 19:34:00.683079 14084 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:00.683126 14084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:00.683249 14084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:34:00.684259 14084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "2d7baf6228f9468da82e6310193869d1"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "e29147cff2c5ac809e248f1363c3329b"
server_key_iv: "34af8dc02d053e61ff0e22210b9d92ae"
server_key_version: "encryptionkey@0"
I20260924 19:34:00.684602 14084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d7baf6228f9468da82e6310193869d1"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "e29147cff2c5ac809e248f1363c3329b"
server_key_iv: "34af8dc02d053e61ff0e22210b9d92ae"
server_key_version: "encryptionkey@0"
I20260924 19:34:00.685824 14084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:00.686532 14099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:00.686690 14084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:00.686771 14084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "2d7baf6228f9468da82e6310193869d1"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "e29147cff2c5ac809e248f1363c3329b"
server_key_iv: "34af8dc02d053e61ff0e22210b9d92ae"
server_key_version: "encryptionkey@0"
I20260924 19:34:00.686832 14084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:00.703011 14084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:00.704937 14084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:00.705083 14084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:00.709162 14084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33011
I20260924 19:34:00.709244 14151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33011 every 8 connection(s)
I20260924 19:34:00.709589 14084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:34:00.710553 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:00.712757 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Bootstrap starting.
I20260924 19:34:00.713274 14152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:00.713500 14152 log.cc:824] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:00.714094 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: No bootstrap required, opened a new log
I20260924 19:34:00.715616 14152 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:00.715773 14152 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:00.715795 14152 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7baf6228f9468da82e6310193869d1, State: Initialized, Role: FOLLOWER
I20260924 19:34:00.715874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 14084
I20260924 19:34:00.715929 14152 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:00.715977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:34:00.716009 14152 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:34:00.716048 14152 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:34:00.716100 14152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:00.716169 392 external_mini_cluster.cc:1428] Setting key c8bb6de5d8ef86aab40ea53949e918b1
I20260924 19:34:00.717037 14152 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:00.717140 14152 leader_election.cc:302] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d7baf6228f9468da82e6310193869d1; no voters:
I20260924 19:34:00.717303 14152 leader_election.cc:288] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:34:00.717325 14157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:34:00.717475 14157 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 1 LEADER]: Becoming Leader. State: Replica: 2d7baf6228f9468da82e6310193869d1, State: Running, Role: LEADER
I20260924 19:34:00.717546 14152 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:34:00.717599 14157 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:00.717948 14157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7baf6228f9468da82e6310193869d1. Latest consensus state: current_term: 1 leader_uuid: "2d7baf6228f9468da82e6310193869d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } } }
I20260924 19:34:00.718147 14157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:00.718091 14159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d7baf6228f9468da82e6310193869d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } } }
I20260924 19:34:00.718374 14159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:00.718516 14166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:34:00.719003 14166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:34:00.720435 14166 catalog_manager.cc:1347] Generated new cluster ID: 08e74a3b0970484287108546c49afb71
I20260924 19:34:00.720487 14166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:34:00 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:00.727130 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:00.716655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.0.98.62:33011)
0924 19:34:00.716823 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:00.716830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:00.716856 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:34:00.716956 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:00.716962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:00.717003 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:00.717111 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:00.717856 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:00.719237 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:00.720284 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:00.720883 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:00.720985 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:00.723712 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:00.723716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:00.723716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:00.723729 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:00.725572 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:00.725954 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:00.725957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:00.725958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:00.726064 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:00.726250 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:00.726251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:00.726251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:00.726543 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:00.726670 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:00.726791 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 19:34:00.726917 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260924 19:34:00.751582 14166 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:34:00.752138 14166 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:34:00.754065 14166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Generated new TSK 0
I20260924 19:34:00.754259 14166 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:34:00.783437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33011
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:00.863258 14178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:00.863461 14178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:00.863484 14178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:00.864619 14178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:00.864694 14178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:00.864723 14178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:34:00.866145 14178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:00.866370 14178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:00.866585 14178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:00.871371 14186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:00.871371 14183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:00.871579 14178 server_base.cc:1034] running on GCE node
W20260924 19:34:00.871507 14184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:00.871834 14178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:00.872037 14178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:00.873181 14178 hybrid_clock.cc:648] HybridClock initialized: now 1790278440873168 us; error 27 us; skew 500 ppm
Sep 24 19:34:00 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:00.875653 14178 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:34:00.876036 14178 webserver.cc:460] Webserver started at http://127.0.98.1:33727/ using document root <none> and password file <none>
I20260924 19:34:00.876222 14178 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:00.876260 14178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:00.876360 14178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:34:00.877373 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "bab04d739a2b4bac9c6090218073e12d"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "1f8611eea2cc84b819aec5fcebb490f0"
server_key_iv: "a5665fd5f08828cdc932266a5c9cf3ba"
server_key_version: "encryptionkey@0"
I20260924 19:34:00.877658 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bab04d739a2b4bac9c6090218073e12d"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "1f8611eea2cc84b819aec5fcebb490f0"
server_key_iv: "a5665fd5f08828cdc932266a5c9cf3ba"
server_key_version: "encryptionkey@0"
I20260924 19:34:00.878796 14178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:34:00.879441 14193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:00.879590 14178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:34:00.879659 14178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "bab04d739a2b4bac9c6090218073e12d"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "1f8611eea2cc84b819aec5fcebb490f0"
server_key_iv: "a5665fd5f08828cdc932266a5c9cf3ba"
server_key_version: "encryptionkey@0"
I20260924 19:34:00.879719 14178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:00.893929 14178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:00.895982 14178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:00.896113 14178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:00.896330 14178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:00.896679 14178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:00.896723 14178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:00.896754 14178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:00.896776 14178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:00.902264 14178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37905
I20260924 19:34:00.902323 14306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37905 every 8 connection(s)
I20260924 19:34:00.902676 14178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:34:00.907895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 14178
I20260924 19:34:00.908018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:34:00.908207 392 external_mini_cluster.cc:1428] Setting key 35ac3bc488e6ae923384efd6c19ebada
Sep 24 19:34:00 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:00.911970 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:00.903397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52961 (local address 127.0.98.62:33011)
0924 19:34:00.903452 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:00.903454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:00.903653 (+ 199us) server_negotiation.cc:407] Connection header received
0924 19:34:00.904161 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:00.904162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:00.904172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:00.904233 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:00.905215 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:00.905727 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:00.906697 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:00.906916 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:00.906991 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:00.909137 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:00.909138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:00.909139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:00.909146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:00.910680 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:00.911055 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:00.911056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:00.911057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:00.911108 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:00.911363 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:00.911364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:00.911365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:00.911478 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:00.911533 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:00.911798 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:34:00.911883 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:34:00.912134 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:00.903364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:52961)
0924 19:34:00.903532 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:00.903574 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:00.904048 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:00.904295 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:00.904298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:00.904924 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:00.905117 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:00.905122 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:00.905793 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:00.905793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:00.906603 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:00.906605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:00.906991 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:00.906992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:00.907082 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:00.907525 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:00.907533 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:00.909054 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:00.910758 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:00.910764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:00.910766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:00.910989 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:00.911180 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:00.911181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:00.911181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:00.911300 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:00.911534 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:00.911540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:00.911667 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 19:34:00.911765 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 19:34:00.911920 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260924 19:34:00.912755 14307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33011
I20260924 19:34:00.912859 14307 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:00.913055 14307 heartbeater.cc:511] Master 127.0.98.62:33011 requested a full tablet report, sending...
I20260924 19:34:00.913604 14116 ts_manager.cc:194] Registered new tserver with Master: bab04d739a2b4bac9c6090218073e12d (127.0.98.1:37905)
I20260924 19:34:00.914191 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52961
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:34:00.931471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33011
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:01.011793 14312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:01.012013 14312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:01.012037 14312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:01.013208 14312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:01.013244 14312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:01.013269 14312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:34:01.014680 14312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:01.014876 14312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:01.015064 14312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:01.020172 14320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.020177 14318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.020166 14317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:01.020434 14312 server_base.cc:1034] running on GCE node
I20260924 19:34:01.020656 14312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:01.020911 14312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:01.022078 14312 hybrid_clock.cc:648] HybridClock initialized: now 1790278441022065 us; error 27 us; skew 500 ppm
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:01.024550 14312 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:34:01.024996 14312 webserver.cc:460] Webserver started at http://127.0.98.2:34169/ using document root <none> and password file <none>
I20260924 19:34:01.025205 14312 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:01.025254 14312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:01.025388 14312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:34:01.026448 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "d7931a294f104c27b20f79546c053bb8"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "99ce2aefff31bdeb9bae00277b3ec97b"
server_key_iv: "78fcd1995762e40f399ea3c09a301254"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.026768 14312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d7931a294f104c27b20f79546c053bb8"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "99ce2aefff31bdeb9bae00277b3ec97b"
server_key_iv: "78fcd1995762e40f399ea3c09a301254"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.028038 14312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:34:01.028757 14327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.028910 14312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:34:01.028992 14312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "d7931a294f104c27b20f79546c053bb8"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "99ce2aefff31bdeb9bae00277b3ec97b"
server_key_iv: "78fcd1995762e40f399ea3c09a301254"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.029058 14312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:01.057097 14312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:01.058925 14312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:01.059060 14312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:01.059283 14312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:01.059581 14312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:01.059619 14312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.059648 14312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:01.059671 14312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.065145 14312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41829
I20260924 19:34:01.065204 14440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41829 every 8 connection(s)
I20260924 19:34:01.065497 14312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:34:01.066342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 14312
I20260924 19:34:01.066442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:34:01.066618 392 external_mini_cluster.cc:1428] Setting key b3e400c5d51b97c1b1842a0d5114e351
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:34:01.074455 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.066244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57049 (local address 127.0.98.62:33011)
0924 19:34:01.066310 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.066312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.066500 (+ 188us) server_negotiation.cc:407] Connection header received
0924 19:34:01.067071 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.067072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.067082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.067152 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:01.068086 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.068573 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.069553 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.069744 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.069798 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.071766 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:01.071768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:01.071768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:01.071777 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:01.073127 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.073532 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.073533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.073534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.073589 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.073810 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.073811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.073812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.073926 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:01.073984 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.074256 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.074352 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:34:01.074607 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.066221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:57049)
0924 19:34:01.066382 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.066424 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.066992 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.067223 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.067228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.067783 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:01.067985 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.067990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.068676 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.068678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.069475 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.069477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.069809 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.069810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.069933 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.070392 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:01.070400 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:01.071680 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:01.073200 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.073208 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.073210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.073462 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.073645 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.073646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.073647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.073754 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.073985 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:01.073992 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:01.074136 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.074226 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.074389 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260924 19:34:01.075098 14441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33011
I20260924 19:34:01.075196 14441 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:01.075376 14441 heartbeater.cc:511] Master 127.0.98.62:33011 requested a full tablet report, sending...
I20260924 19:34:01.075790 14116 ts_manager.cc:194] Registered new tserver with Master: d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
I20260924 19:34:01.076138 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:34:01.091158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33011
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:01.179211 14446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:01.179428 14446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:01.179451 14446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:01.180619 14446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:01.180693 14446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:01.180730 14446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:34:01.182194 14446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:01.182432 14446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:01.182643 14446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:01.187613 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.187657 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.187657 14454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:01.187713 14446 server_base.cc:1034] running on GCE node
I20260924 19:34:01.188052 14446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:01.188261 14446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:01.189393 14446 hybrid_clock.cc:648] HybridClock initialized: now 1790278441189400 us; error 46 us; skew 500 ppm
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:01.191838 14446 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:34:01.192232 14446 webserver.cc:460] Webserver started at http://127.0.98.3:38809/ using document root <none> and password file <none>
I20260924 19:34:01.192443 14446 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:01.192487 14446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:01.192606 14446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:34:01.193679 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "2aa254959def4f169b50fbefb23a40eb"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "c46d9fa20774ae51e5af2eeea52b0e80"
server_key_iv: "d78ff414eaa70c5a8438120739c7b475"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.193966 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2aa254959def4f169b50fbefb23a40eb"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "c46d9fa20774ae51e5af2eeea52b0e80"
server_key_iv: "d78ff414eaa70c5a8438120739c7b475"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.195117 14446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:34:01.195828 14461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.195976 14446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:34:01.196055 14446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "2aa254959def4f169b50fbefb23a40eb"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "c46d9fa20774ae51e5af2eeea52b0e80"
server_key_iv: "d78ff414eaa70c5a8438120739c7b475"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.196118 14446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:01.220191 14446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:01.222024 14446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:01.222158 14446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:01.222368 14446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:01.222697 14446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:01.222728 14446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.222755 14446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:01.222785 14446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.228292 14446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34515
I20260924 19:34:01.228374 14574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34515 every 8 connection(s)
I20260924 19:34:01.228693 14446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:01.235764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 14446
I20260924 19:34:01.235841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:34:01.235988 392 external_mini_cluster.cc:1428] Setting key ee47b5882d5e847bcf8504c48f0124aa
I20260924 19:34:01.237673 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.229389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37979 (local address 127.0.98.62:33011)
0924 19:34:01.229451 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.229453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.229603 (+ 150us) server_negotiation.cc:407] Connection header received
0924 19:34:01.230071 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.230072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.230082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.230135 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:01.231026 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.231513 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.232537 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.232798 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.232846 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.234887 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:01.234889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:01.234889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:01.234896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:01.236067 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.236543 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.236544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.236545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.236610 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.236925 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.236926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.236926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.237141 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:01.237214 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.237522 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.237591 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:34:01.237872 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.229349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:37979)
0924 19:34:01.229493 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.229535 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.229996 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.230193 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.230196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.230752 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:01.230951 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.230957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.231606 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.231607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.232453 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.232455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.232864 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.232865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.233009 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.233452 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:01.233460 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:01.234796 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:01.236158 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.236165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.236167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.236467 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.236706 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.236707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.236707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.236855 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.237247 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:01.237262 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:01.237398 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.237499 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.237632 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260924 19:34:01.238343 14575 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33011
I20260924 19:34:01.238466 14575 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:01.238621 14575 heartbeater.cc:511] Master 127.0.98.62:33011 requested a full tablet report, sending...
I20260924 19:34:01.238948 14116 ts_manager.cc:194] Registered new tserver with Master: 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515)
I20260924 19:34:01.239323 14116 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37979
I20260924 19:34:01.248014 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:34:01.255125 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.249438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.0.98.62:33011)
0924 19:34:01.249527 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.249530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.249596 (+ 66us) server_negotiation.cc:407] Connection header received
0924 19:34:01.249675 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.249676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.249686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.249733 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:01.250289 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.250784 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.251501 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.251682 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.251736 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.252886 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:01.252887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:01.252887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:01.252896 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:01.254100 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.254478 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.254479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.254479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.254546 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.254696 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.254697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.254697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.254839 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:01.254910 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.255000 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.255059 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:34:01.256549 14116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:34:01.258173 14116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:34:01.264533 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.260541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.0.98.2:41829)
0924 19:34:01.261123 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.261126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.261136 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:34:01.261171 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.261173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.261185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.261346 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.261531 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.262491 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.263535 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.264253 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.264324 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.264379 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.264432 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.264472 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":77,"threads_started":1}
I20260924 19:34:01.264518 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.260376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41829 (local address 127.0.0.1:58176)
0924 19:34:01.260513 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.260561 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.260649 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.261421 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.261422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.261424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.261456 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.261458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.262576 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.262577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.263463 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.263465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.264319 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.264320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.264374 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.264377 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.264417 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.264444 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260924 19:34:01.264519 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.260570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34515 (local address 127.0.0.1:59672)
0924 19:34:01.260946 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.260954 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.261004 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.261236 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.261236 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.261238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.261290 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.261291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.262191 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.262192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.263567 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.263569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.264199 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.264200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.264287 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.264302 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.264367 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.264421 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":73,"thread_start_us":323,"threads_started":1}
I20260924 19:34:01.264736 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.260768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.0.98.3:34515)
0924 19:34:01.260921 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.260926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.261011 (+ 85us) server_negotiation.cc:407] Connection header received
0924 19:34:01.261071 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.261074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.261088 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.261157 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.261367 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.262093 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.263646 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.264122 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.264186 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.264258 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.264561 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.264609 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260924 19:34:01.264977 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.260378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37905 (local address 127.0.0.1:50896)
0924 19:34:01.260656 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.260667 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.260718 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.260929 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.260932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.260934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.261001 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.261003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.262278 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.262279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.263521 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.263523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.264766 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.264767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.264827 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.264832 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.264876 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.264916 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":36,"thread_start_us":47,"threads_started":1}
I20260924 19:34:01.265439 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.260484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.0.98.1:37905)
0924 19:34:01.260763 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.260768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.260777 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:34:01.260811 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.260812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.260825 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.260888 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.261079 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.262183 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.263603 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.264254 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.264324 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.264388 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.264912 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.264977 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":67,"threads_started":1}
I20260924 19:34:01.265373 14509 tablet_service.cc:1467] Processing CreateTablet for tablet 885ced6f7d784c2f880a757bcc95a694 (DEFAULT_TABLE table=test-table [id=539cf563b31b416d9a42d751e61dc2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:01.265762 14509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885ced6f7d784c2f880a757bcc95a694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:01.265671 14375 tablet_service.cc:1467] Processing CreateTablet for tablet 885ced6f7d784c2f880a757bcc95a694 (DEFAULT_TABLE table=test-table [id=539cf563b31b416d9a42d751e61dc2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:01.265976 14375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885ced6f7d784c2f880a757bcc95a694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:01.266139 14241 tablet_service.cc:1467] Processing CreateTablet for tablet 885ced6f7d784c2f880a757bcc95a694 (DEFAULT_TABLE table=test-table [id=539cf563b31b416d9a42d751e61dc2f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:01.266407 14241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885ced6f7d784c2f880a757bcc95a694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:01.268965 14592 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Bootstrap starting.
I20260924 19:34:01.269189 14593 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: Bootstrap starting.
I20260924 19:34:01.269654 14592 tablet_bootstrap.cc:654] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:01.269713 14593 tablet_bootstrap.cc:654] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:01.270087 14592 log.cc:824] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:01.270138 14593 log.cc:824] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:01.270437 14594 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Bootstrap starting.
I20260924 19:34:01.270730 14592 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: No bootstrap required, opened a new log
I20260924 19:34:01.270789 14592 ts_tablet_manager.cc:1397] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:34:01.270828 14593 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: No bootstrap required, opened a new log
I20260924 19:34:01.270892 14593 ts_tablet_manager.cc:1397] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:34:01.271067 14594 tablet_bootstrap.cc:654] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:01.271579 14594 log.cc:824] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:01.272490 14592 raft_consensus.cc:348] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.272679 14592 raft_consensus.cc:374] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:01.272742 14592 raft_consensus.cc:729] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab04d739a2b4bac9c6090218073e12d, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.272583 14593 raft_consensus.cc:348] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.272786 14593 raft_consensus.cc:374] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:01.272868 14593 raft_consensus.cc:729] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa254959def4f169b50fbefb23a40eb, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.272884 14592 consensus_queue.cc:260] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.273044 14593 consensus_queue.cc:260] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.273180 14592 ts_tablet_manager.cc:1428] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:34:01.273253 14307 heartbeater.cc:503] Master 127.0.98.62:33011 was elected leader, sending a full tablet report...
I20260924 19:34:01.273280 14594 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: No bootstrap required, opened a new log
I20260924 19:34:01.273344 14594 ts_tablet_manager.cc:1397] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:34:01.273355 14593 ts_tablet_manager.cc:1428] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:34:01.274919 14594 raft_consensus.cc:348] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.275039 14594 raft_consensus.cc:374] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:01.275059 14594 raft_consensus.cc:729] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7931a294f104c27b20f79546c053bb8, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.275138 14594 consensus_queue.cc:260] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.275331 14594 ts_tablet_manager.cc:1428] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:34:01.275379 14441 heartbeater.cc:503] Master 127.0.98.62:33011 was elected leader, sending a full tablet report...
I20260924 19:34:01.275432 14575 heartbeater.cc:503] Master 127.0.98.62:33011 was elected leader, sending a full tablet report...
I20260924 19:34:01.307502 14600 raft_consensus.cc:482] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:01.307613 14600 raft_consensus.cc:504] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.307998 14600 leader_election.cc:288] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bab04d739a2b4bac9c6090218073e12d (127.0.98.1:37905), 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515)
I20260924 19:34:01.310734 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.308290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40261 (local address 127.0.98.1:37905)
0924 19:34:01.308352 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.308354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.308436 (+ 82us) server_negotiation.cc:407] Connection header received
0924 19:34:01.308481 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.308482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.308494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.308538 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.308701 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.309152 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.310058 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.310418 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.310480 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.310490 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.310628 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.310669 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:34:01.310732 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.308260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54561 (local address 127.0.98.3:34515)
0924 19:34:01.308358 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.308360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.308369 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:34:01.308397 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.308397 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.308407 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.308463 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.308608 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.309119 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.310039 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.310410 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.310467 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.310478 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.310624 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.310659 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:34:01.310752 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.308164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37905 (local address 127.0.98.2:40261)
0924 19:34:01.308370 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.308378 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.308428 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.308547 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.308548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.308550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.308617 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.308618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.309223 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.309223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.309973 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.309975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.310496 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.310497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.310570 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.310574 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.310617 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.310654 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":33,"thread_start_us":116,"threads_started":1}
I20260924 19:34:01.310751 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.308159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34515 (local address 127.0.98.2:54561)
0924 19:34:01.308227 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.308238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.308278 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.308482 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.308484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.308485 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.308549 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.308551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.309195 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.309196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.309950 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.309952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.310482 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.310483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.310564 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.310566 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.310610 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.310654 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":36}
I20260924 19:34:01.311162 14261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885ced6f7d784c2f880a757bcc95a694" candidate_uuid: "d7931a294f104c27b20f79546c053bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab04d739a2b4bac9c6090218073e12d" is_pre_election: true
I20260924 19:34:01.311219 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885ced6f7d784c2f880a757bcc95a694" candidate_uuid: "d7931a294f104c27b20f79546c053bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa254959def4f169b50fbefb23a40eb" is_pre_election: true
I20260924 19:34:01.311308 14261 raft_consensus.cc:2454] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7931a294f104c27b20f79546c053bb8 in term 0.
I20260924 19:34:01.311363 14529 raft_consensus.cc:2454] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7931a294f104c27b20f79546c053bb8 in term 0.
I20260924 19:34:01.311530 14331 leader_election.cc:302] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bab04d739a2b4bac9c6090218073e12d, d7931a294f104c27b20f79546c053bb8; no voters:
I20260924 19:34:01.311654 14600 raft_consensus.cc:2781] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:34:01.311712 14600 raft_consensus.cc:482] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:34:01.311730 14600 raft_consensus.cc:3037] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:01.312407 14600 raft_consensus.cc:504] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.312531 14600 leader_election.cc:288] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [CANDIDATE]: Term 1 election: Requested vote from peers bab04d739a2b4bac9c6090218073e12d (127.0.98.1:37905), 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515)
I20260924 19:34:01.312700 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885ced6f7d784c2f880a757bcc95a694" candidate_uuid: "d7931a294f104c27b20f79546c053bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa254959def4f169b50fbefb23a40eb"
I20260924 19:34:01.312711 14261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885ced6f7d784c2f880a757bcc95a694" candidate_uuid: "d7931a294f104c27b20f79546c053bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bab04d739a2b4bac9c6090218073e12d"
I20260924 19:34:01.312778 14529 raft_consensus.cc:3037] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:01.312779 14261 raft_consensus.cc:3037] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:01.313602 14529 raft_consensus.cc:2454] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7931a294f104c27b20f79546c053bb8 in term 1.
I20260924 19:34:01.313601 14261 raft_consensus.cc:2454] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7931a294f104c27b20f79546c053bb8 in term 1.
I20260924 19:34:01.313753 14331 leader_election.cc:302] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bab04d739a2b4bac9c6090218073e12d, d7931a294f104c27b20f79546c053bb8; no voters:
I20260924 19:34:01.313840 14600 raft_consensus.cc:2781] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:34:01.313982 14600 raft_consensus.cc:686] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 LEADER]: Becoming Leader. State: Replica: d7931a294f104c27b20f79546c053bb8, State: Running, Role: LEADER
I20260924 19:34:01.314078 14600 consensus_queue.cc:237] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.314880 14115 catalog_manager.cc:5495] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7931a294f104c27b20f79546c053bb8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d7931a294f104c27b20f79546c053bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } health_report { overall_health: UNKNOWN } } }
W20260924 19:34:01.315948 14442 tablet.cc:2369] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:34:01.321576 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.319582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.0.98.2:41829)
0924 19:34:01.319622 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.319624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.319632 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:34:01.319661 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.319662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.319678 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.319737 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:34:01.319842 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.320295 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.321028 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.321203 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.321247 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.321356 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:34:01.321445 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:34:01.321497 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.321512 (+ 15us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.321528 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:34:01.323199 14529 raft_consensus.cc:1261] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 1 FOLLOWER]: Refusing update from remote peer d7931a294f104c27b20f79546c053bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:34:01.323279 14261 raft_consensus.cc:1261] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Refusing update from remote peer d7931a294f104c27b20f79546c053bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:34:01.323603 14602 consensus_queue.cc:1035] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:34:01.323736 14600 consensus_queue.cc:1035] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:34:01.326131 14607 mvcc.cc:205] Tried to move back new op lower bound from 7332980495657365504 to 7332980495623061504. Current Snapshot: MvccSnapshot[applied={T|T < 7332980495657365504}]
I20260924 19:34:01.327267 14606 mvcc.cc:205] Tried to move back new op lower bound from 7332980495657365504 to 7332980495623061504. Current Snapshot: MvccSnapshot[applied={T|T < 7332980495657365504}]
I20260924 19:34:01.328161 14116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:34:01.328245 14116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:34:01.328922 14116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Sending DeleteTablet for 3 replicas of tablet 885ced6f7d784c2f880a757bcc95a694
I20260924 19:34:01.329216 14375 tablet_service.cc:1514] Processing DeleteTablet for tablet 885ced6f7d784c2f880a757bcc95a694 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58176
I20260924 19:34:01.329236 14509 tablet_service.cc:1514] Processing DeleteTablet for tablet 885ced6f7d784c2f880a757bcc95a694 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59672
I20260924 19:34:01.329439 14241 tablet_service.cc:1514] Processing DeleteTablet for tablet 885ced6f7d784c2f880a757bcc95a694 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50896
I20260924 19:34:01.329646 14616 tablet_replica.cc:330] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: stopping tablet replica
I20260924 19:34:01.329722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 14178
I20260924 19:34:01.329741 14616 raft_consensus.cc:2229] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:01.329838 14616 raft_consensus.cc:2258] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:01.332746 14614 tablet_replica.cc:330] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: stopping tablet replica
I20260924 19:34:01.332854 14614 raft_consensus.cc:2229] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:01.333029 14614 raft_consensus.cc:2258] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:01.333412 14614 ts_tablet_manager.cc:1905] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:01.334887 14614 ts_tablet_manager.cc:1918] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:01.334967 14614 log.cc:1196] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/885ced6f7d784c2f880a757bcc95a694
I20260924 19:34:01.335220 14614 ts_tablet_manager.cc:1939] T 885ced6f7d784c2f880a757bcc95a694 P 2aa254959def4f169b50fbefb23a40eb: Deleting consensus metadata
I20260924 19:34:01.335632 14102 catalog_manager.cc:4841] TS 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515): tablet 885ced6f7d784c2f880a757bcc95a694 (table test-table [id=539cf563b31b416d9a42d751e61dc2f4]) successfully deleted
W20260924 19:34:01.336498 14103 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37905 (error 108)
I20260924 19:34:01.336740 14615 tablet_replica.cc:330] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: stopping tablet replica
I20260924 19:34:01.336768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 14312
I20260924 19:34:01.336905 14615 raft_consensus.cc:2229] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:34:01.337023 14615 raft_consensus.cc:2258] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:01.337041 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.336923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37905 (local address 127.0.0.1:50908)
0924 19:34:01.336975 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.337008 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:34:01.337139 14103 catalog_manager.cc:4584] TS bab04d739a2b4bac9c6090218073e12d (127.0.98.1:37905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 885ced6f7d784c2f880a757bcc95a694: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37905: connect: Connection refused (error 111)
I20260924 19:34:01.341816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 14446
I20260924 19:34:01.341902 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.341794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41829 (local address 127.0.0.1:58190)
0924 19:34:01.341839 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.341872 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 19:34:01.345729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 14084
I20260924 19:34:01.350754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33011
--webserver_interface=127.0.98.62
--webserver_port=40277
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:01.429419 14618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:01.429631 14618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:01.429653 14618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:01.430769 14618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:34:01.430804 14618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:01.430817 14618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:01.430827 14618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:34:01.430838 14618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:34:01.432235 14618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:01.432433 14618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:01.432608 14618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:01.440618 14627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.440606 14625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:01.440831 14618 server_base.cc:1034] running on GCE node
W20260924 19:34:01.440622 14624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:01.441138 14618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:01.441457 14618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:01.442595 14618 hybrid_clock.cc:648] HybridClock initialized: now 1790278441442583 us; error 29 us; skew 500 ppm
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:01.445024 14618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:34:01.445410 14618 webserver.cc:460] Webserver started at http://127.0.98.62:40277/ using document root <none> and password file <none>
I20260924 19:34:01.445595 14618 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:01.445643 14618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:01.446986 14618 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:01.447710 14634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.447929 14618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:01.448020 14618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "2d7baf6228f9468da82e6310193869d1"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "e29147cff2c5ac809e248f1363c3329b"
server_key_iv: "34af8dc02d053e61ff0e22210b9d92ae"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.448443 14618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:01.458040 14618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:01.459736 14618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:01.459846 14618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:01.463519 14618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33011
I20260924 19:34:01.463558 14686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33011 every 8 connection(s)
I20260924 19:34:01.463927 14618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:34:01.465154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 14618
I20260924 19:34:01.465402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37905
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33011
--webserver_port=33727
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:34:01.465523 14687 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:34:01.466145 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Bootstrap starting.
I20260924 19:34:01.473969 14687 log.cc:824] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:01.475817 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:34:01.476065 14687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Bootstrap complete.
I20260924 19:34:01.478210 14687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:01.478482 14687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7baf6228f9468da82e6310193869d1, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.478636 14687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:01.478708 14687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:34:01.478760 14687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:34:01.478803 14687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:34:01.479775 14687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:01.479867 14687 leader_election.cc:302] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d7baf6228f9468da82e6310193869d1; no voters:
I20260924 19:34:01.480041 14687 leader_election.cc:288] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:34:01.480085 14691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:34:01.480199 14691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 2 LEADER]: Becoming Leader. State: Replica: 2d7baf6228f9468da82e6310193869d1, State: Running, Role: LEADER
I20260924 19:34:01.480295 14687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:34:01.480281 14691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:01.480578 14693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7baf6228f9468da82e6310193869d1. Latest consensus state: current_term: 2 leader_uuid: "2d7baf6228f9468da82e6310193869d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } } }
I20260924 19:34:01.480573 14692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2d7baf6228f9468da82e6310193869d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } } }
I20260924 19:34:01.480703 14693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:01.480712 14692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:01.480908 14697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:34:01.481658 14697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:34:01.481889 14697 catalog_manager.cc:1259] Loaded cluster ID: 08e74a3b0970484287108546c49afb71
I20260924 19:34:01.482048 14697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:34:01.482327 14697 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 19:34:01.486905 14697 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 19:34:01.489281 14697 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:34:01.489516 14697 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Loaded TSK: 0
I20260924 19:34:01.489745 14697 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:34:01.546519 14689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:01.546736 14689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:01.546772 14689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:01.547974 14689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:01.548028 14689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:01.548064 14689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:34:01.549733 14689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33727
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:01.549979 14689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:01.550191 14689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:01.555027 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.554978 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.555078 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:01.555300 14689 server_base.cc:1034] running on GCE node
I20260924 19:34:01.555526 14689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:01.555784 14689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:01.556926 14689 hybrid_clock.cc:648] HybridClock initialized: now 1790278441556907 us; error 33 us; skew 500 ppm
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:01.559547 14689 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:34:01.560043 14689 webserver.cc:460] Webserver started at http://127.0.98.1:33727/ using document root <none> and password file <none>
I20260924 19:34:01.560272 14689 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:01.560335 14689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:01.561785 14689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:01.562500 14724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.562668 14689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:01.562752 14689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "bab04d739a2b4bac9c6090218073e12d"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "1f8611eea2cc84b819aec5fcebb490f0"
server_key_iv: "a5665fd5f08828cdc932266a5c9cf3ba"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.563094 14689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:01.581925 14689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:01.583968 14689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:01.584106 14689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:01.584323 14689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:01.584782 14731 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:34:01.586688 14689 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:34:01.586745 14689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:34:01.586781 14689 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:34:01.587415 14689 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:34:01.587448 14689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:01.587550 14731 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Bootstrap starting.
I20260924 19:34:01.593979 14689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37905
I20260924 19:34:01.594359 14689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:34:01.596803 14838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37905 every 8 connection(s)
I20260924 19:34:01.598145 14731 log.cc:824] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:01.600004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 14689
I20260924 19:34:01.600273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41829
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33011
--webserver_port=34169
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:34:01.600795 14731 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:34:01.601049 14731 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Bootstrap complete.
I20260924 19:34:01.602016 14731 ts_tablet_manager.cc:1397] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260924 19:34:01.603008 14731 raft_consensus.cc:348] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.603235 14731 raft_consensus.cc:729] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab04d739a2b4bac9c6090218073e12d, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.603407 14731 consensus_queue.cc:260] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.603678 14731 ts_tablet_manager.cc:1428] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:01.608952 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.594992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:33981)
0924 19:34:01.595174 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.595218 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.595709 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.598977 (+ 3268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.598981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.599326 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:01.599591 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.599598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.601317 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.601319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.602583 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.602586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.603405 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.603406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.603549 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.604073 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:01.604082 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:01.605405 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:01.607103 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.607110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.607112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.607410 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.607686 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.607687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.607687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.607803 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.608253 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:01.608259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:01.608433 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.608521 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.608731 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260924 19:34:01.609853 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.595268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33981 (local address 127.0.98.62:33011)
0924 19:34:01.598506 (+ 3238us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.598517 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.598565 (+ 48us) server_negotiation.cc:407] Connection header received
0924 19:34:01.598709 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.598717 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.598774 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.598926 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:01.599734 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.601182 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.602695 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.603273 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.603845 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.605522 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:01.605526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:01.605527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:01.605536 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:01.607005 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.607504 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.607507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.607507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.607617 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.607869 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.607870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.607871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.608170 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:01.609417 (+ 1247us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.609510 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.609651 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3180,"thread_start_us":79,"threads_started":1}
I20260924 19:34:01.611497 14839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33011
I20260924 19:34:01.611588 14839 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:01.611765 14839 heartbeater.cc:511] Master 127.0.98.62:33011 requested a full tablet report, sending...
I20260924 19:34:01.612538 14651 ts_manager.cc:194] Registered new tserver with Master: bab04d739a2b4bac9c6090218073e12d (127.0.98.1:37905)
I20260924 19:34:01.613700 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33981
I20260924 19:34:01.614756 14839 heartbeater.cc:503] Master 127.0.98.62:33011 was elected leader, sending a full tablet report...
I20260924 19:34:01.617972 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.614071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37905 (local address 127.0.0.1:50918)
0924 19:34:01.614238 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.614274 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.614344 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.615281 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.615285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.615287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.615374 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.615376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.616161 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.616162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.617084 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.617087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.617684 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.617685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.617778 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.617788 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.617851 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.617901 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260924 19:34:01.617970 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.614959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.0.98.1:37905)
0924 19:34:01.615105 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.615109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.615119 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:34:01.615152 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.615154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.615167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.615433 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.615468 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.616083 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.617157 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.617608 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.617669 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.617723 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.617870 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.617909 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260924 19:34:01.618562 14759 tablet_service.cc:1514] Processing DeleteTablet for tablet 885ced6f7d784c2f880a757bcc95a694 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50918
I20260924 19:34:01.619029 14848 tablet_replica.cc:330] stopping tablet replica
I20260924 19:34:01.619128 14848 raft_consensus.cc:2229] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:01.619196 14848 raft_consensus.cc:2258] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:01.619485 14848 ts_tablet_manager.cc:1905] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:01.621369 14848 ts_tablet_manager.cc:1918] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:01.621431 14848 log.cc:1196] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/885ced6f7d784c2f880a757bcc95a694
I20260924 19:34:01.621621 14848 ts_tablet_manager.cc:1939] T 885ced6f7d784c2f880a757bcc95a694 P bab04d739a2b4bac9c6090218073e12d: Deleting consensus metadata
I20260924 19:34:01.622080 14638 catalog_manager.cc:4841] TS bab04d739a2b4bac9c6090218073e12d (127.0.98.1:37905): tablet 885ced6f7d784c2f880a757bcc95a694 (table test-table [id=539cf563b31b416d9a42d751e61dc2f4]) successfully deleted
W20260924 19:34:01.686700 14844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:01.686904 14844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:01.686928 14844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:01.688040 14844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:01.688076 14844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:01.688100 14844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:34:01.689531 14844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34169
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:01.689723 14844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:01.689896 14844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:01.694507 14856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.694561 14854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.694571 14853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:01.694737 14844 server_base.cc:1034] running on GCE node
I20260924 19:34:01.694912 14844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:01.695118 14844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:01.696255 14844 hybrid_clock.cc:648] HybridClock initialized: now 1790278441696240 us; error 26 us; skew 500 ppm
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:01.701733 14844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:34:01.702158 14844 webserver.cc:460] Webserver started at http://127.0.98.2:34169/ using document root <none> and password file <none>
I20260924 19:34:01.702370 14844 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:01.702433 14844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:01.703745 14844 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:01.704347 14863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.704493 14844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:34:01.704573 14844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "d7931a294f104c27b20f79546c053bb8"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "99ce2aefff31bdeb9bae00277b3ec97b"
server_key_iv: "78fcd1995762e40f399ea3c09a301254"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.704932 14844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:01.713184 14844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:01.714970 14844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:01.715106 14844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:01.715309 14844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:01.715700 14870 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:34:01.717252 14844 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:34:01.717309 14844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:34:01.717343 14844 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:34:01.718001 14844 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:34:01.718034 14844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:01.718106 14870 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Bootstrap starting.
I20260924 19:34:01.724459 14844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41829
I20260924 19:34:01.724550 14977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41829 every 8 connection(s)
I20260924 19:34:01.724906 14844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:34:01.726696 14870 log.cc:824] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:01.728047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 14844
I20260924 19:34:01.728312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34515
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33011
--webserver_port=38809
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:34:01.729169 14870 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:34:01.729507 14870 tablet_bootstrap.cc:492] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Bootstrap complete.
I20260924 19:34:01.730376 14870 ts_tablet_manager.cc:1397] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:01.731621 14870 raft_consensus.cc:348] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.731901 14870 raft_consensus.cc:729] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7931a294f104c27b20f79546c053bb8, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.732065 14870 consensus_queue.cc:260] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } }
I20260924 19:34:01.732349 14870 ts_tablet_manager.cc:1428] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 19:34:01.734689 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.725626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53925 (local address 127.0.98.62:33011)
0924 19:34:01.725695 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.725698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.725917 (+ 219us) server_negotiation.cc:407] Connection header received
0924 19:34:01.726414 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.726416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.726429 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.726488 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:01.727220 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.727722 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.728925 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.729142 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.729197 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.731704 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:01.731706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:01.731706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:01.731715 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:01.733281 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.733740 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.733742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.733742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.733809 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.734023 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.734024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.734024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.734167 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:01.734232 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.734502 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.734595 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:34:01.734740 14980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.725562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:53925)
0924 19:34:01.725808 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.725841 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.726317 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.726556 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.726560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.726863 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:01.727135 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.727142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.727822 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.727823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.728851 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.728854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.729225 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.729226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.729354 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.729893 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:01.729903 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:01.731610 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:01.733393 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.733400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.733403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.733660 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.733874 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.733875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.733875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.733970 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.734231 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:01.734235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:01.734381 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.734465 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.734594 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260924 19:34:01.735210 14978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33011
I20260924 19:34:01.735316 14978 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:01.735517 14978 heartbeater.cc:511] Master 127.0.98.62:33011 requested a full tablet report, sending...
I20260924 19:34:01.736140 14651 ts_manager.cc:194] Registered new tserver with Master: d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
I20260924 19:34:01.736793 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53925
I20260924 19:34:01.737675 14978 heartbeater.cc:503] Master 127.0.98.62:33011 was elected leader, sending a full tablet report...
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 19:34:01.744297 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.736528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.0.98.2:41829)
0924 19:34:01.736704 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.736709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.736718 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:34:01.736746 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.736747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.736761 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.736828 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:01.737310 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.738053 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.739024 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.739299 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.739357 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.741580 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:01.741588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:01.741589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:01.741596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:01.743101 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.743520 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.743523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.743524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.743578 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.743743 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.743743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.743744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.743892 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:01.743962 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.744122 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.744208 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":121,"threads_started":1}
I20260924 19:34:01.744297 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.736454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41829 (local address 127.0.0.1:40522)
0924 19:34:01.736516 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.736526 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.736566 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.736838 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.736839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.737198 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:01.737237 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.737239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.738130 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.738131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.738954 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.738957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.739364 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.739366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.739421 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.739946 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:01.739951 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:01.741479 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:01.743195 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.743204 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.743206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.743441 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.743635 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.743636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.743636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.743693 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.743956 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:01.743963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:01.744017 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.744081 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.744186 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 19:34:01.744697 14912 tablet_service.cc:1514] Processing DeleteTablet for tablet 885ced6f7d784c2f880a757bcc95a694 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40522
I20260924 19:34:01.745216 14985 tablet_replica.cc:330] stopping tablet replica
I20260924 19:34:01.745329 14985 raft_consensus.cc:2229] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:01.745388 14985 raft_consensus.cc:2258] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:01.745730 14985 ts_tablet_manager.cc:1905] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:01.748061 14985 ts_tablet_manager.cc:1918] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:01.748136 14985 log.cc:1196] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/885ced6f7d784c2f880a757bcc95a694
I20260924 19:34:01.748338 14985 ts_tablet_manager.cc:1939] T 885ced6f7d784c2f880a757bcc95a694 P d7931a294f104c27b20f79546c053bb8: Deleting consensus metadata
I20260924 19:34:01.748943 14637 catalog_manager.cc:4841] TS d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829): tablet 885ced6f7d784c2f880a757bcc95a694 (table test-table [id=539cf563b31b416d9a42d751e61dc2f4]) successfully deleted
W20260924 19:34:01.811036 14982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:01.811246 14982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:01.811268 14982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:01.812407 14982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:01.812453 14982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:01.812479 14982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:34:01.813961 14982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38809
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:01.814188 14982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:01.814421 14982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:01.819208 14990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:01.819200 14993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:01.819258 14982 server_base.cc:1034] running on GCE node
W20260924 19:34:01.819200 14991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:01.819717 14982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:01.819931 14982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:01.821081 14982 hybrid_clock.cc:648] HybridClock initialized: now 1790278441821070 us; error 26 us; skew 500 ppm
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:01.823489 14982 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:34:01.823896 14982 webserver.cc:460] Webserver started at http://127.0.98.3:38809/ using document root <none> and password file <none>
I20260924 19:34:01.824097 14982 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:01.824155 14982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:01.825460 14982 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:01.825985 15000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.826125 14982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:34:01.826205 14982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "2aa254959def4f169b50fbefb23a40eb"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "c46d9fa20774ae51e5af2eeea52b0e80"
server_key_iv: "d78ff414eaa70c5a8438120739c7b475"
server_key_version: "encryptionkey@0"
I20260924 19:34:01.826572 14982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:01.838984 14982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:01.840679 14982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:01.840811 14982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:01.841012 14982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:01.841284 14982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:01.841323 14982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.841353 14982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:01.841377 14982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.846764 14982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34515
I20260924 19:34:01.846829 15113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34515 every 8 connection(s)
I20260924 19:34:01.847129 14982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:34:01 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:01.853681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 14982
I20260924 19:34:01.856074 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.847779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48327 (local address 127.0.98.62:33011)
0924 19:34:01.847846 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.847848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.848077 (+ 229us) server_negotiation.cc:407] Connection header received
0924 19:34:01.848563 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.848564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.848573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.848649 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:01.849633 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.850124 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.851080 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.851264 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.851320 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.853314 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:01.853316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:01.853316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:01.853324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:01.854792 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.855234 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.855235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.855235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.855302 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.855500 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.855501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.855501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.855617 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:01.855686 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.855910 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.855987 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:34:01.856248 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.847782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:48327)
0924 19:34:01.847930 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.847995 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.848480 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.848729 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.848732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.849336 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:01.849560 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.849565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.850189 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.850189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.851014 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.851016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.851323 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.851324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.851420 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.851877 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:01.851886 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:01.853220 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:01.854869 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.854874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.854876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.855166 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.855360 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:01.855360 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:01.855360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:01.855455 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:01.855676 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:01.855679 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:01.855785 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.855867 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.856037 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260924 19:34:01.856698 15114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33011
I20260924 19:34:01.856791 15114 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:01.856973 15114 heartbeater.cc:511] Master 127.0.98.62:33011 requested a full tablet report, sending...
I20260924 19:34:01.857344 14651 ts_manager.cc:194] Registered new tserver with Master: 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515)
I20260924 19:34:01.857678 14651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48327
I20260924 19:34:01.859929 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.854025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.0.98.62:33011)
0924 19:34:01.854135 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.854137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.854298 (+ 161us) server_negotiation.cc:407] Connection header received
0924 19:34:01.854378 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.854379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.854387 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.854437 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:01.855040 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.855679 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.856487 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.856777 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.856836 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.857923 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:01.857924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:01.857924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:01.857929 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:01.859040 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.859333 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.859334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.859334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.859398 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:01.859537 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:01.859538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:01.859538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:01.859644 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:01.859764 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.859794 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.859858 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
I20260924 19:34:01.861124 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:34:01.863772 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.861539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.0.98.62:33011)
0924 19:34:01.861589 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.861591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.861681 (+ 90us) server_negotiation.cc:407] Connection header received
0924 19:34:01.861749 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.861750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.861784 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.861840 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:34:01.861947 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.862386 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.863112 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.863308 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.863358 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.863502 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:34:01.863591 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:34:01.863657 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.863693 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.863712 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:34:01.864040 14651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:34:01.865656 14651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:34:01.867462 14759 tablet_service.cc:1467] Processing CreateTablet for tablet accd78fe0e334196998b9f979fc9e682 (DEFAULT_TABLE table=test-table [id=1804b86a040a4c558b1100161569eab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:01.867635 14759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accd78fe0e334196998b9f979fc9e682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:01.868898 14731 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Bootstrap starting.
I20260924 19:34:01.869097 14912 tablet_service.cc:1467] Processing CreateTablet for tablet accd78fe0e334196998b9f979fc9e682 (DEFAULT_TABLE table=test-table [id=1804b86a040a4c558b1100161569eab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:01.869190 14731 tablet_bootstrap.cc:654] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:01.869266 14912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accd78fe0e334196998b9f979fc9e682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:01.869668 14731 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: No bootstrap required, opened a new log
I20260924 19:34:01.869710 14731 ts_tablet_manager.cc:1397] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:01.869819 14731 raft_consensus.cc:348] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.869872 14731 raft_consensus.cc:374] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:01.869899 14731 raft_consensus.cc:729] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab04d739a2b4bac9c6090218073e12d, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.869933 14731 consensus_queue.cc:260] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.870028 14731 ts_tablet_manager.cc:1428] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.870651 14870 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: Bootstrap starting.
I20260924 19:34:01.870918 14870 tablet_bootstrap.cc:654] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:01.871353 14870 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: No bootstrap required, opened a new log
I20260924 19:34:01.871393 14870 ts_tablet_manager.cc:1397] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:01.871690 14870 raft_consensus.cc:348] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.871749 14870 raft_consensus.cc:374] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:01.871775 14870 raft_consensus.cc:729] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7931a294f104c27b20f79546c053bb8, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.871814 14870 consensus_queue.cc:260] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.871902 14870 ts_tablet_manager.cc:1428] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:01.872272 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.867241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34515 (local address 127.0.0.1:37478)
0924 19:34:01.867284 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.867295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.867338 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.869133 (+ 1795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.869134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.869136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.869181 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.869183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.870452 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.870453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.871304 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.871307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.872032 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.872033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.872101 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.872106 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.872162 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.872203 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:34:01.872288 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.868722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37478 (local address 127.0.98.3:34515)
0924 19:34:01.868980 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.868985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.868996 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:34:01.869032 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.869034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.869049 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.869125 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.869247 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.870331 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.871383 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.871966 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.872033 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.872094 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.872173 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.872203 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260924 19:34:01.873013 15048 tablet_service.cc:1467] Processing CreateTablet for tablet accd78fe0e334196998b9f979fc9e682 (DEFAULT_TABLE table=test-table [id=1804b86a040a4c558b1100161569eab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:01.873301 15048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accd78fe0e334196998b9f979fc9e682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:01.875943 15123 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: Bootstrap starting.
I20260924 19:34:01.876497 15123 tablet_bootstrap.cc:654] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:01.877004 15123 log.cc:824] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:01.877635 15123 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: No bootstrap required, opened a new log
I20260924 19:34:01.877691 15123 ts_tablet_manager.cc:1397] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:34:01.879009 15123 raft_consensus.cc:348] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.879146 15123 raft_consensus.cc:374] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:01.879177 15123 raft_consensus.cc:729] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa254959def4f169b50fbefb23a40eb, State: Initialized, Role: FOLLOWER
I20260924 19:34:01.879292 15123 consensus_queue.cc:260] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.879522 15123 ts_tablet_manager.cc:1428] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:34:01.879558 15114 heartbeater.cc:503] Master 127.0.98.62:33011 was elected leader, sending a full tablet report...
W20260924 19:34:01.975569 14979 tablet.cc:2369] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:34:01.990625 14845 raft_consensus.cc:482] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:01.990770 14845 raft_consensus.cc:504] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.991098 14845 leader_election.cc:288] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515), d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
I20260924 19:34:01.994832 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.991346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34515 (local address 127.0.98.1:56899)
0924 19:34:01.991407 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.991419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.991468 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.991628 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.991629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.991631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.991694 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.991696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.992568 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.992569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.993860 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.993863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.994591 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.994592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.994649 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.994653 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.994707 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.994752 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 19:34:01.994843 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.991415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56899 (local address 127.0.98.3:34515)
0924 19:34:01.991470 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.991472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.991519 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:34:01.991547 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.991547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.991557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.991610 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.991789 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.992468 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.993952 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.994506 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.994579 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.994591 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.994730 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.994773 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:34:01.995000 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.991460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41829 (local address 127.0.98.1:60927)
0924 19:34:01.991648 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:01.991657 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:01.991728 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:01.991953 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:01.991954 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:01.991956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:01.992029 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.992030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.992865 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.992866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.994004 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:01.994006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.994747 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.994748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:01.994831 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.994834 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:34:01.994885 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:34:01.994920 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":108,"threads_started":1}
I20260924 19:34:01.995002 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:01.991563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60927 (local address 127.0.98.2:41829)
0924 19:34:01.991621 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:01.991623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:01.991793 (+ 170us) server_negotiation.cc:407] Connection header received
0924 19:34:01.991820 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:01.991822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:01.991848 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:01.991912 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:01.992098 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.992775 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.994074 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:01.994654 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:01.994707 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:01.994798 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:01.994897 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:34:01.994940 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:34:01.995185 15068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accd78fe0e334196998b9f979fc9e682" candidate_uuid: "bab04d739a2b4bac9c6090218073e12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa254959def4f169b50fbefb23a40eb" is_pre_election: true
I20260924 19:34:01.995406 15068 raft_consensus.cc:2454] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab04d739a2b4bac9c6090218073e12d in term 0.
I20260924 19:34:01.995633 14727 leader_election.cc:302] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa254959def4f169b50fbefb23a40eb, bab04d739a2b4bac9c6090218073e12d; no voters:
I20260924 19:34:01.995700 14932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accd78fe0e334196998b9f979fc9e682" candidate_uuid: "bab04d739a2b4bac9c6090218073e12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7931a294f104c27b20f79546c053bb8" is_pre_election: true
I20260924 19:34:01.995756 14845 raft_consensus.cc:2781] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:34:01.995803 14845 raft_consensus.cc:482] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:34:01.995817 14845 raft_consensus.cc:3037] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:01.995852 14932 raft_consensus.cc:2454] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bab04d739a2b4bac9c6090218073e12d in term 0.
I20260924 19:34:01.996515 14845 raft_consensus.cc:504] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.996658 14845 leader_election.cc:288] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 1 election: Requested vote from peers 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515), d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
I20260924 19:34:01.996825 14932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accd78fe0e334196998b9f979fc9e682" candidate_uuid: "bab04d739a2b4bac9c6090218073e12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7931a294f104c27b20f79546c053bb8"
I20260924 19:34:01.996845 15068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accd78fe0e334196998b9f979fc9e682" candidate_uuid: "bab04d739a2b4bac9c6090218073e12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa254959def4f169b50fbefb23a40eb"
I20260924 19:34:01.996922 14932 raft_consensus.cc:3037] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:01.996919 15068 raft_consensus.cc:3037] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:01.997603 15068 raft_consensus.cc:2454] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab04d739a2b4bac9c6090218073e12d in term 1.
I20260924 19:34:01.997637 14932 raft_consensus.cc:2454] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bab04d739a2b4bac9c6090218073e12d in term 1.
I20260924 19:34:01.997759 14727 leader_election.cc:302] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa254959def4f169b50fbefb23a40eb, bab04d739a2b4bac9c6090218073e12d; no voters:
I20260924 19:34:01.997877 14845 raft_consensus.cc:2781] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:34:01.998047 14845 raft_consensus.cc:686] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 LEADER]: Becoming Leader. State: Replica: bab04d739a2b4bac9c6090218073e12d, State: Running, Role: LEADER
I20260924 19:34:01.998113 14845 consensus_queue.cc:237] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:01.998915 14650 catalog_manager.cc:5495] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d reported cstate change: term changed from 0 to 1, leader changed from <none> to bab04d739a2b4bac9c6090218073e12d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bab04d739a2b4bac9c6090218073e12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } health_report { overall_health: UNKNOWN } } }
I20260924 19:34:02.016122 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.014022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58978 (local address 127.0.98.1:37905)
0924 19:34:02.014073 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:02.014075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:02.014082 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:34:02.014117 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:02.014118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:02.014136 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:02.014192 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:34:02.014280 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:02.014750 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:02.015456 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:02.015648 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:02.015723 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:02.015813 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:34:02.015936 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:34:02.015992 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:02.016023 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 19:34:02.016046 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:34:02.017669 14932 raft_consensus.cc:1261] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Refusing update from remote peer bab04d739a2b4bac9c6090218073e12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:34:02.017710 15068 raft_consensus.cc:1261] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 1 FOLLOWER]: Refusing update from remote peer bab04d739a2b4bac9c6090218073e12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:34:02.017985 14845 consensus_queue.cc:1035] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:34:02.018117 15127 consensus_queue.cc:1035] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:34:02.020043 15130 mvcc.cc:205] Tried to move back new op lower bound from 7332980498502090752 to 7332980498424815616. Current Snapshot: MvccSnapshot[applied={T|T < 7332980498502090752}]
I20260924 19:34:02.022221 14651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:34:02.022315 14651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:34:02.022990 14651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Sending DeleteTablet for 3 replicas of tablet accd78fe0e334196998b9f979fc9e682
I20260924 19:34:02.023310 14759 tablet_service.cc:1514] Processing DeleteTablet for tablet accd78fe0e334196998b9f979fc9e682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50918
I20260924 19:34:02.023307 14912 tablet_service.cc:1514] Processing DeleteTablet for tablet accd78fe0e334196998b9f979fc9e682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40522
I20260924 19:34:02.023315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 14689
I20260924 19:34:02.023392 14848 tablet_replica.cc:330] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: stopping tablet replica
I20260924 19:34:02.023430 14848 raft_consensus.cc:2229] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:34:02.023504 14848 raft_consensus.cc:2258] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:02.023559 14985 tablet_replica.cc:330] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: stopping tablet replica
I20260924 19:34:02.023599 14985 raft_consensus.cc:2229] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:02.023639 14985 raft_consensus.cc:2258] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:02.024175 15048 tablet_service.cc:1514] Processing DeleteTablet for tablet accd78fe0e334196998b9f979fc9e682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37478
I20260924 19:34:02.024318 14985 ts_tablet_manager.cc:1905] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:02.024385 15139 tablet_replica.cc:330] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: stopping tablet replica
I20260924 19:34:02.024480 15139 raft_consensus.cc:2229] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:02.024572 15139 raft_consensus.cc:2258] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:02.025781 14985 ts_tablet_manager.cc:1918] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:02.025830 14985 log.cc:1196] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/accd78fe0e334196998b9f979fc9e682
I20260924 19:34:02.025990 14985 ts_tablet_manager.cc:1939] T accd78fe0e334196998b9f979fc9e682 P d7931a294f104c27b20f79546c053bb8: Deleting consensus metadata
I20260924 19:34:02.026250 14637 catalog_manager.cc:4841] TS d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829): tablet accd78fe0e334196998b9f979fc9e682 (table test-table [id=1804b86a040a4c558b1100161569eab2]) successfully deleted
W20260924 19:34:02.028362 14638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37905 (error 108)
I20260924 19:34:02.028685 15139 ts_tablet_manager.cc:1905] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:02.028831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 14844
I20260924 19:34:02.029314 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.028883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37905 (local address 127.0.0.1:58982)
0924 19:34:02.029243 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.029282 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331}
W20260924 19:34:02.029744 14638 catalog_manager.cc:4584] TS bab04d739a2b4bac9c6090218073e12d (127.0.98.1:37905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet accd78fe0e334196998b9f979fc9e682: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37905: connect: Connection refused (error 111)
I20260924 19:34:02.030817 15139 ts_tablet_manager.cc:1918] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:02.030890 15139 log.cc:1196] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/accd78fe0e334196998b9f979fc9e682
I20260924 19:34:02.031118 15139 ts_tablet_manager.cc:1939] T accd78fe0e334196998b9f979fc9e682 P 2aa254959def4f169b50fbefb23a40eb: Deleting consensus metadata
I20260924 19:34:02.031507 14637 catalog_manager.cc:4841] TS 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515): tablet accd78fe0e334196998b9f979fc9e682 (table test-table [id=1804b86a040a4c558b1100161569eab2]) successfully deleted
I20260924 19:34:02.033464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 14982
I20260924 19:34:02.038177 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 14618
I20260924 19:34:02.042557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33011
--webserver_interface=127.0.98.62
--webserver_port=40277
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:02.120671 15141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:02.120891 15141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:02.120914 15141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:02.122113 15141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:34:02.122155 15141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:02.122169 15141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:02.122179 15141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:34:02.122189 15141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:34:02.123669 15141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:02.123901 15141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:02.124130 15141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:02.128816 15149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:02.128871 15147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:02.128835 15146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:02.128913 15141 server_base.cc:1034] running on GCE node
I20260924 19:34:02.129220 15141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:02.129513 15141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:02.130667 15141 hybrid_clock.cc:648] HybridClock initialized: now 1790278442130652 us; error 28 us; skew 500 ppm
Sep 24 19:34:02 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:02.133028 15141 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:34:02.133458 15141 webserver.cc:460] Webserver started at http://127.0.98.62:40277/ using document root <none> and password file <none>
I20260924 19:34:02.133652 15141 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:02.133699 15141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:02.134934 15141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:02.135491 15156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.135640 15141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.135720 15141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "2d7baf6228f9468da82e6310193869d1"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "e29147cff2c5ac809e248f1363c3329b"
server_key_iv: "34af8dc02d053e61ff0e22210b9d92ae"
server_key_version: "encryptionkey@0"
I20260924 19:34:02.136070 15141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:02.145766 15141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:02.147707 15141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:02.147833 15141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:02.151547 15141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33011
I20260924 19:34:02.151594 15208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33011 every 8 connection(s)
I20260924 19:34:02.151930 15141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:34:02.153594 15209 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:34:02.154291 15209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Bootstrap starting.
I20260924 19:34:02.156870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 15141
I20260924 19:34:02.157171 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37905
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33011
--webserver_port=33727
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:34:02.162822 15209 log.cc:824] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:02.165167 15209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:34:02.165444 15209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1: Bootstrap complete.
I20260924 19:34:02.167501 15209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:02.167726 15209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7baf6228f9468da82e6310193869d1, State: Initialized, Role: FOLLOWER
I20260924 19:34:02.167886 15209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:02.167963 15209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:34:02.168011 15209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:34:02.168058 15209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:34:02.169095 15209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:02.169193 15209 leader_election.cc:302] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d7baf6228f9468da82e6310193869d1; no voters:
I20260924 19:34:02.169356 15209 leader_election.cc:288] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:34:02.169404 15213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:34:02.169525 15213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [term 3 LEADER]: Becoming Leader. State: Replica: 2d7baf6228f9468da82e6310193869d1, State: Running, Role: LEADER
I20260924 19:34:02.169636 15209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:34:02.169608 15213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } }
I20260924 19:34:02.170001 15214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2d7baf6228f9468da82e6310193869d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } } }
I20260924 19:34:02.170085 15214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:02.170462 15220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:34:02.171339 15220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:34:02.171446 15220 catalog_manager.cc:1259] Loaded cluster ID: 08e74a3b0970484287108546c49afb71
I20260924 19:34:02.171563 15220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:34:02.170213 15215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7baf6228f9468da82e6310193869d1. Latest consensus state: current_term: 3 leader_uuid: "2d7baf6228f9468da82e6310193869d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7baf6228f9468da82e6310193869d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33011 } } }
I20260924 19:34:02.171671 15215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7baf6228f9468da82e6310193869d1 [sys.catalog]: This master's current role is: LEADER
F20260924 19:34:02.171855 15220 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4e790029ad google::LogMessage::Fail() at ??:0
@ 0x7f4e7900588a google::LogMessage::SendToLog() at ??:0
@ 0x7f4e790024de google::LogMessage::Flush() at ??:0
@ 0x7f4e79003a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4e80f280f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f4e80f318b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4e7aed00d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4e7aec5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e7bd296db start_thread at ??:0
@ 0x7f4e77b0061f clone at ??:0
W20260924 19:34:02.252534 15211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:02.252768 15211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:02.252794 15211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:02.254035 15211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:02.254081 15211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:02.254109 15211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:34:02.255604 15211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33727
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:02.255864 15211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:02.256111 15211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:02.261112 15248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:02.261185 15211 server_base.cc:1034] running on GCE node
W20260924 19:34:02.261282 15246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:02.261098 15245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:02.261549 15211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:02.261803 15211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:02.262949 15211 hybrid_clock.cc:648] HybridClock initialized: now 1790278442262930 us; error 34 us; skew 500 ppm
Sep 24 19:34:02 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:02.265296 15211 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:34:02.265859 15211 webserver.cc:460] Webserver started at http://127.0.98.1:33727/ using document root <none> and password file <none>
I20260924 19:34:02.266126 15211 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:02.266185 15211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:02.267582 15211 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:02.268162 15255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.268308 15211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.268380 15211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "bab04d739a2b4bac9c6090218073e12d"
format_stamp: "Formatted at 2026-09-24 19:34:00 on dist-test-slave-m8jb"
server_key: "1f8611eea2cc84b819aec5fcebb490f0"
server_key_iv: "a5665fd5f08828cdc932266a5c9cf3ba"
server_key_version: "encryptionkey@0"
I20260924 19:34:02.268759 15211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:02.290751 15211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:02.292968 15211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:02.293092 15211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:02.293316 15211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:02.293735 15262 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:34:02.295706 15211 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:34:02.295753 15211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:34:02.295785 15211 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:34:02.296406 15211 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:34:02.296435 15211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:02.296528 15262 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Bootstrap starting.
I20260924 19:34:02.303007 15211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37905
I20260924 19:34:02.303390 15211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:34:02.304819 15369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37905 every 8 connection(s)
I20260924 19:34:02.306921 15262 log.cc:824] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:02.309737 15262 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:34:02.310057 15262 tablet_bootstrap.cc:492] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Bootstrap complete.
I20260924 19:34:02.311151 15262 ts_tablet_manager.cc:1397] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260924 19:34:02.312549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 15211
I20260924 19:34:02.312801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41829
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33011
--webserver_port=34169
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:34:02.312655 15262 raft_consensus.cc:348] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:02.313005 15262 raft_consensus.cc:729] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab04d739a2b4bac9c6090218073e12d, State: Initialized, Role: FOLLOWER
I20260924 19:34:02.313153 15262 consensus_queue.cc:260] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:02.313455 15262 ts_tablet_manager.cc:1428] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260924 19:34:02.381593 15372 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:34:02.304089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:41449)
0924 19:34:02.304284 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.304321 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:02.304864 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:02.381388 (+ 76524us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
W20260924 19:34:02.381840 15370 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33011 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33011: Client connection negotiation failed: client connection to 127.0.98.62:33011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:34:02.382170 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.382059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:43505)
0924 19:34:02.382109 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.382147 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 19:34:02.382457 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.382383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:51961)
0924 19:34:02.382420 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.382441 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:34:02.382599 15370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:34:02.408437 15375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:02.408674 15375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:02.408710 15375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:02.409886 15375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:02.409924 15375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:02.409948 15375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:34:02.411352 15375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34169
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:02.411545 15375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:02.411733 15375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:02.416426 15383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:02.416440 15380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:02.416453 15375 server_base.cc:1034] running on GCE node
W20260924 19:34:02.416512 15381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:02.416853 15375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:02.417062 15375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:02.418207 15375 hybrid_clock.cc:648] HybridClock initialized: now 1790278442418194 us; error 28 us; skew 500 ppm
Sep 24 19:34:02 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:02.420598 15375 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:34:02.421018 15375 webserver.cc:460] Webserver started at http://127.0.98.2:34169/ using document root <none> and password file <none>
I20260924 19:34:02.421208 15375 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:02.421247 15375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:02.422422 15375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:02.422941 15390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.423086 15375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:34:02.423161 15375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "d7931a294f104c27b20f79546c053bb8"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "99ce2aefff31bdeb9bae00277b3ec97b"
server_key_iv: "78fcd1995762e40f399ea3c09a301254"
server_key_version: "encryptionkey@0"
I20260924 19:34:02.423484 15375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:02.443509 15375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:02.445425 15375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:02.445535 15375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:02.445734 15375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:02.446029 15375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:02.446061 15375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.446082 15375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:02.446105 15375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.451514 15375 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41829
I20260924 19:34:02.451596 15503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41829 every 8 connection(s)
I20260924 19:34:02.451901 15375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:34:02.452916 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.452530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:54119)
0924 19:34:02.452700 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.452773 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
W20260924 19:34:02.453277 15504 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33011 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33011: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
I20260924 19:34:02.453521 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.453429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:43359)
0924 19:34:02.453479 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.453503 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 19:34:02.453714 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.453659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:41417)
0924 19:34:02.453686 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.453703 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 19:34:02.453785 15504 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:34:02.458253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 15375
I20260924 19:34:02.458493 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34515
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33011
--webserver_port=38809
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:02.537786 15507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:02.538005 15507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:02.538028 15507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:02.539176 15507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:02.539218 15507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:02.539247 15507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:34:02.540719 15507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38809
--tserver_master_addrs=127.0.98.62:33011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:02.540935 15507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:02.541138 15507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:02.547012 15513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:02.548395 15515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:02.548409 15512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:02.549383 15507 server_base.cc:1034] running on GCE node
I20260924 19:34:02.549598 15507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:02.549818 15507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:02.550951 15507 hybrid_clock.cc:648] HybridClock initialized: now 1790278442550938 us; error 36 us; skew 500 ppm
Sep 24 19:34:02 dist-test-slave-m8jb krb5kdc[14071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:02.553748 15507 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:34:02.554184 15507 webserver.cc:460] Webserver started at http://127.0.98.3:38809/ using document root <none> and password file <none>
I20260924 19:34:02.554387 15507 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:02.554435 15507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:02.555757 15507 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:02.556392 15522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.556556 15507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:34:02.556659 15507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "2aa254959def4f169b50fbefb23a40eb"
format_stamp: "Formatted at 2026-09-24 19:34:01 on dist-test-slave-m8jb"
server_key: "c46d9fa20774ae51e5af2eeea52b0e80"
server_key_iv: "d78ff414eaa70c5a8438120739c7b475"
server_key_version: "encryptionkey@0"
I20260924 19:34:02.557041 15507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:02.573773 15507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:02.575748 15507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:02.575882 15507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:02.576094 15507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:02.576391 15507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:02.576426 15507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.576455 15507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:02.576478 15507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:02.582320 15507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34515
I20260924 19:34:02.582394 15635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34515 every 8 connection(s)
I20260924 19:34:02.582708 15507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:34:02.583786 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.583334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:52703)
0924 19:34:02.583497 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.583591 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
W20260924 19:34:02.584233 15636 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33011 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33011: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
I20260924 19:34:02.584520 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.584426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:45881)
0924 19:34:02.584469 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.584497 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:34:02.584796 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:02.584723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:36315)
0924 19:34:02.584758 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:02.584780 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 19:34:02.584889 15636 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:34:02.587984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 15507
I20260924 19:34:03.383262 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:03.382960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:56683)
0924 19:34:03.383130 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:03.383177 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260924 19:34:03.454398 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:03.454082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:51867)
0924 19:34:03.454252 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:03.454320 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":59,"threads_started":1}
I20260924 19:34:03.585500 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:03.585233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:41423)
0924 19:34:03.585391 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:03.585442 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260924 19:34:03.964690 15643 raft_consensus.cc:482] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:03.964847 15643 raft_consensus.cc:504] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:03.965180 15643 leader_election.cc:288] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515), d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
Sep 24 19:34:03 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 19:34:03 dist-test-slave-m8jb krb5kdc[14071](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 19:34:03.976577 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:03.965721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43569 (local address 127.0.98.2:41829)
0924 19:34:03.965906 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:03.965922 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:03.965965 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:34:03.966625 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:03.966630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:03.966692 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:03.966869 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:03.967591 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:03.968548 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:03.969750 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:03.970206 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:03.970300 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:03.973008 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:03.973015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:03.973015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:03.973034 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:03.974810 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:03.975279 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:03.975283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:03.975284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:03.975378 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:03.975638 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:03.975639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:03.975639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:03.975908 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:03.975996 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:03.976302 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 19:34:03.976453 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260924 19:34:03.976567 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:03.965370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34515 (local address 127.0.98.1:33179)
0924 19:34:03.965527 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:03.965541 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:03.965654 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:03.966647 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:03.966651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:03.967196 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:03.967440 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:03.967449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:03.968691 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:03.968692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:03.969657 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:03.969660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:03.970330 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:03.970331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:03.970426 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:03.971054 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:03.971064 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:03.973189 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:03.974981 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:03.974983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:03.974983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:03.975229 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:03.975501 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:03.975502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:03.975502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:03.975578 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:03.975986 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:03.975987 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:03.976127 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 19:34:03.976228 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:34:03.976453 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":96,"thread_start_us":69,"threads_started":1}
I20260924 19:34:03.976594 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:03.965516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33179 (local address 127.0.98.3:34515)
0924 19:34:03.965702 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:03.965715 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:03.965762 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:34:03.966339 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:03.966344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:03.966403 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:03.966552 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:03.967576 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:03.968548 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:03.969760 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:03.970240 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:03.970364 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:03.973309 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:03.973315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:03.973316 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:03.973328 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:03.974882 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:03.975317 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:03.975320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:03.975321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:03.975420 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:03.975647 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:03.975648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:03.975648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:03.975911 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:03.975990 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:03.976304 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 19:34:03.976458 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":56,"threads_started":1}
I20260924 19:34:03.976907 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:03.965558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41829 (local address 127.0.98.1:43569)
0924 19:34:03.965720 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:03.965729 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:03.965777 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:03.966867 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:03.966870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:03.967203 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:03.967441 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:03.967452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:03.968680 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:03.968681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:03.969659 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:03.969662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:03.970300 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:03.970302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:03.970424 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:03.971090 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:03.971095 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:03.972826 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:03.974902 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:03.974911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:03.974913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:03.975195 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:03.975444 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:03.975445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:03.975446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:03.975567 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:03.975979 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:03.975986 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:03.976126 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 19:34:03.976225 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:34:03.976836 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":58,"thread_start_us":52,"threads_started":1}
W20260924 19:34:03.977774 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
W20260924 19:34:03.977865 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
I20260924 19:34:03.977883 15258 leader_election.cc:302] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab04d739a2b4bac9c6090218073e12d; no voters: 2aa254959def4f169b50fbefb23a40eb, d7931a294f104c27b20f79546c053bb8
I20260924 19:34:03.977988 15643 raft_consensus.cc:2726] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:34:04.383975 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:04.383816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:34945)
0924 19:34:04.383887 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:04.383931 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 19:34:04.455135 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:04.454907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:56363)
0924 19:34:04.455038 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:04.455089 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260924 19:34:04.586268 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:04.586052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:58405)
0924 19:34:04.586184 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:04.586225 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260924 19:34:05.384872 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:05.384571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:52387)
0924 19:34:05.384746 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:05.384793 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260924 19:34:05.455839 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:05.455626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:51165)
0924 19:34:05.455751 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:05.455790 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260924 19:34:05.586184 15652 raft_consensus.cc:482] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:05.586267 15652 raft_consensus.cc:504] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:05.586392 15652 leader_election.cc:288] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515), d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
W20260924 19:34:05.586796 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
W20260924 19:34:05.586930 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
I20260924 19:34:05.586968 15258 leader_election.cc:302] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab04d739a2b4bac9c6090218073e12d; no voters: 2aa254959def4f169b50fbefb23a40eb, d7931a294f104c27b20f79546c053bb8
I20260924 19:34:05.586980 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:05.586806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:48853)
0924 19:34:05.586906 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:05.586945 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260924 19:34:05.587033 15652 raft_consensus.cc:2726] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:34:06.385824 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:06.385511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:37875)
0924 19:34:06.385692 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:06.385760 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260924 19:34:06.456693 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:06.456358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:39101)
0924 19:34:06.456532 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:06.456596 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
I20260924 19:34:06.587774 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:06.587496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:59987)
0924 19:34:06.587668 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:06.587720 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
I20260924 19:34:07.386703 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:07.386436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:45541)
0924 19:34:07.386609 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:07.386653 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260924 19:34:07.457504 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:07.457202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:55485)
0924 19:34:07.457388 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:07.457433 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260924 19:34:07.460289 15659 raft_consensus.cc:482] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:07.460430 15659 raft_consensus.cc:504] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:07.460592 15659 leader_election.cc:288] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515), d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
W20260924 19:34:07.461095 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
W20260924 19:34:07.461199 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
I20260924 19:34:07.461221 15258 leader_election.cc:302] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab04d739a2b4bac9c6090218073e12d; no voters: 2aa254959def4f169b50fbefb23a40eb, d7931a294f104c27b20f79546c053bb8
I20260924 19:34:07.461304 15659 raft_consensus.cc:2726] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:34:07.588668 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:07.588357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:47557)
0924 19:34:07.588537 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:07.588592 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260924 19:34:08.387631 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:08.387320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:56423)
0924 19:34:08.387519 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:08.387583 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":114,"threads_started":1}
I20260924 19:34:08.458289 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:08.458011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:55417)
0924 19:34:08.458186 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:08.458230 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20260924 19:34:08.589344 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:08.589142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:55567)
0924 19:34:08.589261 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:08.589292 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260924 19:34:09.388447 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:09.388189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:60723)
0924 19:34:09.388320 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:09.388364 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260924 19:34:09.458977 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:09.458737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:46429)
0924 19:34:09.458884 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:09.458928 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260924 19:34:09.589944 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:09.589737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:36851)
0924 19:34:09.589864 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:09.589893 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260924 19:34:10.174669 15667 raft_consensus.cc:482] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:10.174777 15667 raft_consensus.cc:504] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:10.174914 15667 leader_election.cc:288] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515), d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
W20260924 19:34:10.175271 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
W20260924 19:34:10.175333 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
I20260924 19:34:10.175354 15258 leader_election.cc:302] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab04d739a2b4bac9c6090218073e12d; no voters: 2aa254959def4f169b50fbefb23a40eb, d7931a294f104c27b20f79546c053bb8
I20260924 19:34:10.175447 15667 raft_consensus.cc:2726] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:34:10.389074 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:10.388880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:47011)
0924 19:34:10.388994 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:10.389031 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260924 19:34:10.459722 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:10.459463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:47195)
0924 19:34:10.459614 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:10.459667 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260924 19:34:10.590577 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:10.590366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:36073)
0924 19:34:10.590499 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:10.590533 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260924 19:34:11.389827 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:11.389606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:41397)
0924 19:34:11.389741 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:11.389782 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260924 19:34:11.460444 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:11.460223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:43865)
0924 19:34:11.460350 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:11.460396 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260924 19:34:11.591270 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:11.591057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:35525)
0924 19:34:11.591188 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:11.591227 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260924 19:34:12.390513 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:12.390256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:46609)
0924 19:34:12.390428 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:12.390466 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":60,"threads_started":1}
I20260924 19:34:12.461172 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:12.460953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:40457)
0924 19:34:12.461085 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:12.461130 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260924 19:34:12.591933 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:12.591697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:38785)
0924 19:34:12.591847 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:12.591882 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260924 19:34:13.196031 15677 raft_consensus.cc:482] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:13.196132 15677 raft_consensus.cc:504] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2aa254959def4f169b50fbefb23a40eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34515 } } peers { permanent_uuid: "bab04d739a2b4bac9c6090218073e12d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "d7931a294f104c27b20f79546c053bb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41829 } }
I20260924 19:34:13.196286 15677 leader_election.cc:288] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515), d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829)
W20260924 19:34:13.196789 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7931a294f104c27b20f79546c053bb8 (127.0.98.2:41829): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
W20260924 19:34:13.196867 15258 leader_election.cc:341] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2aa254959def4f169b50fbefb23a40eb (127.0.98.3:34515): Not found: Tablet not found: accd78fe0e334196998b9f979fc9e682
I20260924 19:34:13.196893 15258 leader_election.cc:302] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bab04d739a2b4bac9c6090218073e12d; no voters: 2aa254959def4f169b50fbefb23a40eb, d7931a294f104c27b20f79546c053bb8
I20260924 19:34:13.196997 15677 raft_consensus.cc:2726] T accd78fe0e334196998b9f979fc9e682 P bab04d739a2b4bac9c6090218073e12d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:34:13.391191 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:13.390997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:43231)
0924 19:34:13.391110 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:13.391145 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 19:34:13.461854 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:13.461628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:33177)
0924 19:34:13.461756 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:13.461806 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260924 19:34:13.592597 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:13.592371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:40525)
0924 19:34:13.592512 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:13.592552 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260924 19:34:14.391997 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:14.391703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:57395)
0924 19:34:14.391882 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:14.391932 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260924 19:34:14.462688 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:14.462404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:37117)
0924 19:34:14.462580 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:14.462637 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260924 19:34:14.593484 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:14.593165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:42467)
0924 19:34:14.593347 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:14.593391 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":94,"threads_started":1}
I20260924 19:34:15.392814 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:15.392543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:34081)
0924 19:34:15.392710 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:15.392758 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
I20260924 19:34:15.463438 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:15.463192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:34547)
0924 19:34:15.463332 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:15.463376 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260924 19:34:15.594293 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:15.594025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:43221)
0924 19:34:15.594201 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:15.594245 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260924 19:34:16.393801 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:16.393463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:40583)
0924 19:34:16.393691 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:16.393753 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
I20260924 19:34:16.464291 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:16.463960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:57003)
0924 19:34:16.464153 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:16.464226 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260924 19:34:16.595126 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:16.594844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:50243)
0924 19:34:16.595025 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:16.595081 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260924 19:34:17.394588 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:17.394369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.1:32915)
0924 19:34:17.394497 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:17.394535 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260924 19:34:17.465099 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:17.464894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.2:51609)
0924 19:34:17.465016 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:17.465053 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260924 19:34:17.595919 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:17.595680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33011 (local address 127.0.98.3:52681)
0924 19:34:17.595836 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:17.595876 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260924 19:34:17.598352 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 15211
I20260924 19:34:17.602617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 15375
I20260924 19:34:17.606251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 15507
2026-09-24T19:34:17Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18078 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:34:17 dist-test-slave-m8jb krb5kdc[15695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:34:17 dist-test-slave-m8jb krb5kdc[15695](info): set up 2 sockets
Sep 24 19:34:17 dist-test-slave-m8jb krb5kdc[15695](info): commencing operation
krb5kdc: starting...
W20260924 19:34:18.564999 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:34:18 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:34:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:34:18.677567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:18.756319 15708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:18.756546 15708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:18.756582 15708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:18.758002 15708 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:34:18.758044 15708 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:34:18.758055 15708 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:34:18.758065 15708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:34:18.758077 15708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:18.758109 15708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:18.758132 15708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:34:18.758152 15708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:34:18.759665 15708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:18.759902 15708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:18.760128 15708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:18.764858 15716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:18.764853 15708 server_base.cc:1034] running on GCE node
W20260924 19:34:18.764834 15713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:18.764948 15714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:18.765326 15708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:18.765601 15708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:18.766741 15708 hybrid_clock.cc:648] HybridClock initialized: now 1790278458766723 us; error 33 us; skew 500 ppm
Sep 24 19:34:18 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:18.769141 15708 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:34:18.769551 15708 webserver.cc:460] Webserver started at http://127.0.98.62:45933/ using document root <none> and password file <none>
I20260924 19:34:18.769757 15708 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:18.769804 15708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:18.769923 15708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:34:18.771016 15708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "bbdb8780f9a8456884d843009e67c5c3"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "7f3a508358fa78680d41dc0201bf5916"
server_key_iv: "3068d9c48edfd045bca05f56a8949a5e"
server_key_version: "encryptionkey@0"
I20260924 19:34:18.771340 15708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbdb8780f9a8456884d843009e67c5c3"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "7f3a508358fa78680d41dc0201bf5916"
server_key_iv: "3068d9c48edfd045bca05f56a8949a5e"
server_key_version: "encryptionkey@0"
I20260924 19:34:18.772578 15708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:34:18.773288 15723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:18.773437 15708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:34:18.773518 15708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "bbdb8780f9a8456884d843009e67c5c3"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "7f3a508358fa78680d41dc0201bf5916"
server_key_iv: "3068d9c48edfd045bca05f56a8949a5e"
server_key_version: "encryptionkey@0"
I20260924 19:34:18.773581 15708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:18.782585 15708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:18.784323 15708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:18.784451 15708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:18.787976 15708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40655
I20260924 19:34:18.788015 15775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40655 every 8 connection(s)
I20260924 19:34:18.788328 15708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:34:18.789196 15776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:18.791404 15776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Bootstrap starting.
I20260924 19:34:18.791774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 15708
I20260924 19:34:18.791863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:34:18.791971 15776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:18.792040 392 external_mini_cluster.cc:1428] Setting key 55107aa972d05242276bf6282b95733c
I20260924 19:34:18.792233 15776 log.cc:824] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:18.792937 15776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: No bootstrap required, opened a new log
I20260924 19:34:18.794960 15776 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:34:18.795138 15776 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:18.795181 15776 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdb8780f9a8456884d843009e67c5c3, State: Initialized, Role: FOLLOWER
I20260924 19:34:18.795307 15776 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:34:18.795379 15776 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:34:18.795413 15776 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:34:18.795456 15776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:18.796069 15776 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:34:18.796173 15776 leader_election.cc:302] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbdb8780f9a8456884d843009e67c5c3; no voters:
I20260924 19:34:18.796314 15776 leader_election.cc:288] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:34:18.796348 15781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:34:18.796496 15781 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 1 LEADER]: Becoming Leader. State: Replica: bbdb8780f9a8456884d843009e67c5c3, State: Running, Role: LEADER
I20260924 19:34:18.796593 15776 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:34:18.796658 15781 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:34:18.797334 15783 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbdb8780f9a8456884d843009e67c5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } } }
I20260924 19:34:18.797410 15783 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:18.797515 15783 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbdb8780f9a8456884d843009e67c5c3. Latest consensus state: current_term: 1 leader_uuid: "bbdb8780f9a8456884d843009e67c5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } } }
I20260924 19:34:18.797556 15783 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:18.797745 15790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 19:34:18 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:18.798203 15790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:34:18.800428 15790 catalog_manager.cc:1347] Generated new cluster ID: 0c8cddbf8efa49a5ac0f25f59790144f
I20260924 19:34:18.800482 15790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:34:18.802093 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:18.792498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.0.98.62:40655)
0924 19:34:18.792689 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:18.792699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:18.792733 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:34:18.793150 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:18.793173 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:18.793221 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:18.793331 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:18.794049 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:18.794904 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:18.795889 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:18.796346 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:18.796790 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:18.799024 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:18.799028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:18.799028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:18.799039 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:18.800776 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:18.801071 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:18.801074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:18.801075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:18.801164 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:18.801331 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:18.801332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:18.801332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:18.801556 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:18.801683 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:18.801795 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 19:34:18.801919 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260924 19:34:18.811198 15790 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:34:18.811908 15790 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:34:18.820039 15790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Generated new TSK 0
I20260924 19:34:18.820237 15790 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:34:18.852095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40655
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:18.932775 15802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:18.932977 15802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:18.933000 15802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:18.934183 15802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:18.934226 15802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:18.934252 15802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:34:18.935683 15802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:18.935897 15802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:18.936101 15802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:18.940809 15808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:18.940857 15807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:18.940992 15802 server_base.cc:1034] running on GCE node
W20260924 19:34:18.940814 15810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:18.941200 15802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:18.941421 15802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:18.942577 15802 hybrid_clock.cc:648] HybridClock initialized: now 1790278458942562 us; error 27 us; skew 500 ppm
Sep 24 19:34:18 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:18.944942 15802 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:34:18.945358 15802 webserver.cc:460] Webserver started at http://127.0.98.1:38651/ using document root <none> and password file <none>
I20260924 19:34:18.945544 15802 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:18.945587 15802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:18.945726 15802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:34:18.946718 15802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "a95db9885af04cc5a5dfa61fc10e6804"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "1618d3909acc7ff7a8e8c8f8db381b7b"
server_key_iv: "050adb6c3f4483bf78213d992cb4fd1f"
server_key_version: "encryptionkey@0"
I20260924 19:34:18.947017 15802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a95db9885af04cc5a5dfa61fc10e6804"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "1618d3909acc7ff7a8e8c8f8db381b7b"
server_key_iv: "050adb6c3f4483bf78213d992cb4fd1f"
server_key_version: "encryptionkey@0"
I20260924 19:34:18.948191 15802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:34:18.948822 15817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:18.948943 15802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:18.949016 15802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "a95db9885af04cc5a5dfa61fc10e6804"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "1618d3909acc7ff7a8e8c8f8db381b7b"
server_key_iv: "050adb6c3f4483bf78213d992cb4fd1f"
server_key_version: "encryptionkey@0"
I20260924 19:34:18.949079 15802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:18.970607 15802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:18.972394 15802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:18.972512 15802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:18.972750 15802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:18.973047 15802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:18.973078 15802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:18.973098 15802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:18.973121 15802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:18.978682 15802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33601
I20260924 19:34:18.978747 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33601 every 8 connection(s)
I20260924 19:34:18.979032 15802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:34:18 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:18.986899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 15802
I20260924 19:34:18.987008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:34:18.987231 392 external_mini_cluster.cc:1428] Setting key 3c32f9bab0e655dd82c2e2d2f1123151
I20260924 19:34:18.988176 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:18.979710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44797 (local address 127.0.98.62:40655)
0924 19:34:18.979789 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:18.979791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:18.979936 (+ 145us) server_negotiation.cc:407] Connection header received
0924 19:34:18.980410 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:18.980411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:18.980420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:18.980468 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:18.981429 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:18.981938 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:18.983317 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:18.983608 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:18.983663 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:18.985734 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:18.985736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:18.985736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:18.985743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:18.986835 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:18.987261 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:18.987262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:18.987263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:18.987326 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:18.987542 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:18.987543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:18.987543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:18.987709 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:18.987772 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:18.988039 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 19:34:18.988113 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:34:18.988397 15933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:18.979671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:44797)
0924 19:34:18.979817 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:18.979855 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:18.980333 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:18.980522 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:18.980525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:18.981155 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:18.981361 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:18.981365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:18.982045 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:18.982046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:18.983227 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:18.983230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:18.983685 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:18.983686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:18.983786 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:18.984221 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:18.984230 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:18.985644 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:18.986905 (+ 1261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:18.986913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:18.986915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:18.987196 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:18.987373 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:18.987373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:18.987374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:18.987479 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:18.987767 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:18.987771 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:18.987911 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 19:34:18.988019 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:34:18.988186 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260924 19:34:18.989581 15931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40655
I20260924 19:34:18.989691 15931 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:18.989878 15931 heartbeater.cc:511] Master 127.0.98.62:40655 requested a full tablet report, sending...
I20260924 19:34:18.990347 15740 ts_manager.cc:194] Registered new tserver with Master: a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601)
I20260924 19:34:18.990932 15740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44797
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:34:19.010999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40655
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:19.090085 15936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:19.090286 15936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:19.090315 15936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:19.091480 15936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:19.091579 15936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:19.091614 15936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:34:19.093068 15936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:19.093287 15936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:19.093499 15936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:19.098356 15942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:19.098410 15944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:19.098448 15936 server_base.cc:1034] running on GCE node
W20260924 19:34:19.098354 15941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:19.098768 15936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:19.098958 15936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:19.100088 15936 hybrid_clock.cc:648] HybridClock initialized: now 1790278459100074 us; error 29 us; skew 500 ppm
Sep 24 19:34:19 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:19.102316 15936 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:34:19.102720 15936 webserver.cc:460] Webserver started at http://127.0.98.2:45261/ using document root <none> and password file <none>
I20260924 19:34:19.102926 15936 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:19.102972 15936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:19.103093 15936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:34:19.104099 15936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "f8405079a669489d90f494f0fbd3bac0"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "4f643a7248df7fcde2aed66a0a864afd"
server_key_iv: "9be175050e98cd6e0eacd84caaf8b19d"
server_key_version: "encryptionkey@0"
I20260924 19:34:19.104421 15936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f8405079a669489d90f494f0fbd3bac0"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "4f643a7248df7fcde2aed66a0a864afd"
server_key_iv: "9be175050e98cd6e0eacd84caaf8b19d"
server_key_version: "encryptionkey@0"
I20260924 19:34:19.105701 15936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:34:19.106343 15951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:19.106494 15936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:19.106571 15936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "f8405079a669489d90f494f0fbd3bac0"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "4f643a7248df7fcde2aed66a0a864afd"
server_key_iv: "9be175050e98cd6e0eacd84caaf8b19d"
server_key_version: "encryptionkey@0"
I20260924 19:34:19.106635 15936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:19.121426 15936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:19.123397 15936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:19.123530 15936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:19.123740 15936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:19.124017 15936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:19.124048 15936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:19.124076 15936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:19.124099 15936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:19.129807 15936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39579
I20260924 19:34:19.129863 16064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39579 every 8 connection(s)
I20260924 19:34:19.130151 15936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:34:19.135638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 15936
I20260924 19:34:19.135733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:34:19.135905 392 external_mini_cluster.cc:1428] Setting key 654e105862f555e7c884fc4020ac60d7
Sep 24 19:34:19 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:19.139348 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.130907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38349 (local address 127.0.98.62:40655)
0924 19:34:19.130966 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.130968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.131143 (+ 175us) server_negotiation.cc:407] Connection header received
0924 19:34:19.131661 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.131662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.131672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.131716 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:19.132656 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.133144 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.134196 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.134366 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.134409 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.136505 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:19.136507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:19.136507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:19.136514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:19.138118 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:19.138473 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:19.138473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:19.138474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:19.138539 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:19.138753 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:19.138753 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:19.138753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:19.138885 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:19.138975 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.139180 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.139256 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:34:19.139421 16067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.130857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:38349)
0924 19:34:19.131024 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.131063 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:19.131582 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:19.131778 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:19.131781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:19.132334 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:19.132559 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.132564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.133235 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.133236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.134122 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.134125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.134462 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.134463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.134566 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.134999 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:19.135006 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:19.136423 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:19.138197 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:19.138203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:19.138205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:19.138401 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:19.138599 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:19.138600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:19.138600 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:19.138699 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:19.138962 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:19.138964 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:19.139068 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 19:34:19.139141 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:34:19.139258 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260924 19:34:19.139838 16065 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40655
I20260924 19:34:19.139922 16065 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:19.140076 16065 heartbeater.cc:511] Master 127.0.98.62:40655 requested a full tablet report, sending...
I20260924 19:34:19.140446 15740 ts_manager.cc:194] Registered new tserver with Master: f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
I20260924 19:34:19.140836 15740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38349
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:34:19.163118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40655
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:19.242659 16070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:19.242873 16070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:19.242897 16070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:19.244035 16070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:19.244074 16070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:19.244100 16070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:34:19.245606 16070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:19.245805 16070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:19.246006 16070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:19.251096 16076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:19.251214 16070 server_base.cc:1034] running on GCE node
W20260924 19:34:19.251128 16078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:19.251101 16075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:19.251576 16070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:19.251839 16070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:19.252997 16070 hybrid_clock.cc:648] HybridClock initialized: now 1790278459252987 us; error 28 us; skew 500 ppm
Sep 24 19:34:19 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:19.255312 16070 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:34:19.255775 16070 webserver.cc:460] Webserver started at http://127.0.98.3:42615/ using document root <none> and password file <none>
I20260924 19:34:19.255977 16070 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:19.256019 16070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:19.256142 16070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:34:19.257211 16070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "fc42a386b1ef48c1a29e09f81d238b7b"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "92dcfa653bd644c30979b3769587c1da"
server_key_iv: "87ddabbf8c6f99035dbe3bed744ccab6"
server_key_version: "encryptionkey@0"
I20260924 19:34:19.257556 16070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fc42a386b1ef48c1a29e09f81d238b7b"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "92dcfa653bd644c30979b3769587c1da"
server_key_iv: "87ddabbf8c6f99035dbe3bed744ccab6"
server_key_version: "encryptionkey@0"
I20260924 19:34:19.258764 16070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:34:19.259382 16085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:19.259526 16070 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:34:19.259599 16070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "fc42a386b1ef48c1a29e09f81d238b7b"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "92dcfa653bd644c30979b3769587c1da"
server_key_iv: "87ddabbf8c6f99035dbe3bed744ccab6"
server_key_version: "encryptionkey@0"
I20260924 19:34:19.259670 16070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:19.267808 16070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:19.269574 16070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:19.269711 16070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:19.269914 16070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:19.270200 16070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:19.270231 16070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:19.270258 16070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:19.270282 16070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:19.275826 16070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33313
I20260924 19:34:19.275895 16198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33313 every 8 connection(s)
I20260924 19:34:19.276268 16070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:34:19.277562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 16070
I20260924 19:34:19.277649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:34:19.277832 392 external_mini_cluster.cc:1428] Setting key b8f6d04f11fc6ee92353995cbfadebf0
Sep 24 19:34:19 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:19.285365 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.277081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38043 (local address 127.0.98.62:40655)
0924 19:34:19.277147 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.277150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.277309 (+ 159us) server_negotiation.cc:407] Connection header received
0924 19:34:19.277877 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.277878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.277890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.277938 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:19.279074 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.279830 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.280841 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.281021 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.281070 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.283020 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:19.283022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:19.283022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:19.283029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:19.284261 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:19.284603 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:19.284604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:19.284604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:19.284674 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:19.284868 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:19.284869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:19.284869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:19.284968 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:19.285015 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.285241 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.285301 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:34:19.285597 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.276991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:38043)
0924 19:34:19.277175 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.277226 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:19.277784 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:19.278040 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:19.278043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:19.278704 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:19.278979 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.278985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.279914 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.279915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.280776 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.280779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.281077 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.281077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.281173 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.281627 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:19.281636 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:19.282938 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:19.284327 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:19.284333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:19.284334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:19.284544 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:19.284722 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:19.284722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:19.284723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:19.284823 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:19.285012 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:19.285015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:19.285145 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 19:34:19.285224 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:34:19.285385 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260924 19:34:19.286018 16199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40655
I20260924 19:34:19.286103 16199 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:19.286247 16199 heartbeater.cc:511] Master 127.0.98.62:40655 requested a full tablet report, sending...
I20260924 19:34:19.286561 15740 ts_manager.cc:194] Registered new tserver with Master: fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313)
I20260924 19:34:19.286886 15740 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38043
I20260924 19:34:19.289428 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:34:19.295635 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.290676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.0.98.62:40655)
0924 19:34:19.290727 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.290728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.290829 (+ 101us) server_negotiation.cc:407] Connection header received
0924 19:34:19.290870 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.290871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.290878 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.290921 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:19.291395 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.291911 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.292590 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.292820 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.292861 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.293857 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:19.293858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:19.293858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:19.293864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:19.294846 (+ 982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:19.295100 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:19.295101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:19.295101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:19.295140 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:19.295264 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:19.295265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:19.295265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:19.295383 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:19.295452 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.295524 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.295576 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:34:19.297098 15740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:34:19.298853 15740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:34:19.304697 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.300907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60456 (local address 127.0.98.3:33313)
0924 19:34:19.301194 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.301198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.301206 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:34:19.301234 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.301235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.301247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.301309 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:19.301468 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.302337 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.303587 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.304283 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.304341 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.304401 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.304545 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.304587 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":79,"threads_started":1}
I20260924 19:34:19.304684 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.300804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33313 (local address 127.0.0.1:60456)
0924 19:34:19.300949 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.300990 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:19.301052 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:19.301327 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:19.301330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:19.301332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:19.301395 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.301396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.302423 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.302424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.303496 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.303498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.304356 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.304357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.304447 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.304460 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:34:19.304522 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:34:19.304575 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":47,"threads_started":1}
I20260924 19:34:19.304823 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.300979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39579 (local address 127.0.0.1:40614)
0924 19:34:19.301146 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.301155 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:19.301196 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:19.301444 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:19.301445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:19.301446 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:19.301482 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.301483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.302883 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.302884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.303799 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.303801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.304583 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.304584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.304680 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.304684 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:34:19.304715 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 19:34:19.304754 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":36,"thread_start_us":44,"threads_started":1}
I20260924 19:34:19.305075 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.300813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33601 (local address 127.0.0.1:35918)
0924 19:34:19.301064 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.301074 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:19.301306 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:19.301575 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:19.301576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:19.301578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:19.301619 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.301621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.302906 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.302907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.303999 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.304001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.304898 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.304899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.304954 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.304957 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:34:19.304985 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 19:34:19.305018 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":24,"thread_start_us":43,"threads_started":1}
I20260924 19:34:19.305137 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.301177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40614 (local address 127.0.98.2:39579)
0924 19:34:19.301309 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.301313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.301321 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:34:19.301359 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.301361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.301376 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.301448 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:19.301645 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.302763 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.303869 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.304489 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.304933 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.305002 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.305024 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.305063 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260924 19:34:19.305575 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.300987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35918 (local address 127.0.98.1:33601)
0924 19:34:19.301424 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.301428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.301436 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:34:19.301473 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.301475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.301487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.301691 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:19.301761 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.302814 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.304088 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.304813 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.305367 (+ 554us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.305434 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.305459 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.305499 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":76,"threads_started":1}
I20260924 19:34:19.305697 15999 tablet_service.cc:1467] Processing CreateTablet for tablet d28c2779a7814145b43a31803fd8fe31 (DEFAULT_TABLE table=test-table [id=1719137a2d5140b09e6f535f77d47ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:19.305744 16133 tablet_service.cc:1467] Processing CreateTablet for tablet d28c2779a7814145b43a31803fd8fe31 (DEFAULT_TABLE table=test-table [id=1719137a2d5140b09e6f535f77d47ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:19.306051 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28c2779a7814145b43a31803fd8fe31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:19.306054 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28c2779a7814145b43a31803fd8fe31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:19.306150 15865 tablet_service.cc:1467] Processing CreateTablet for tablet d28c2779a7814145b43a31803fd8fe31 (DEFAULT_TABLE table=test-table [id=1719137a2d5140b09e6f535f77d47ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:19.306438 15865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d28c2779a7814145b43a31803fd8fe31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:19.308508 16213 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Bootstrap starting.
I20260924 19:34:19.309250 16213 tablet_bootstrap.cc:654] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:19.309536 16215 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Bootstrap starting.
I20260924 19:34:19.309536 16214 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Bootstrap starting.
I20260924 19:34:19.309783 16213 log.cc:824] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:19.310175 16215 tablet_bootstrap.cc:654] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:19.310187 16214 tablet_bootstrap.cc:654] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:19.310462 16213 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: No bootstrap required, opened a new log
I20260924 19:34:19.310523 16213 ts_tablet_manager.cc:1397] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:34:19.310752 16215 log.cc:824] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:19.310787 16214 log.cc:824] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:19.311450 16215 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: No bootstrap required, opened a new log
I20260924 19:34:19.311518 16215 ts_tablet_manager.cc:1397] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:34:19.311697 16214 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: No bootstrap required, opened a new log
I20260924 19:34:19.311791 16214 ts_tablet_manager.cc:1397] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:34:19.312243 16213 raft_consensus.cc:348] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.312398 16213 raft_consensus.cc:374] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:19.312433 16213 raft_consensus.cc:729] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc42a386b1ef48c1a29e09f81d238b7b, State: Initialized, Role: FOLLOWER
I20260924 19:34:19.312625 16213 consensus_queue.cc:260] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.312929 16213 ts_tablet_manager.cc:1428] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:34:19.313047 16199 heartbeater.cc:503] Master 127.0.98.62:40655 was elected leader, sending a full tablet report...
I20260924 19:34:19.313535 16215 raft_consensus.cc:348] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.313711 16215 raft_consensus.cc:374] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:19.313750 16215 raft_consensus.cc:729] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95db9885af04cc5a5dfa61fc10e6804, State: Initialized, Role: FOLLOWER
I20260924 19:34:19.313714 16214 raft_consensus.cc:348] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.313854 16214 raft_consensus.cc:374] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:19.313856 16215 consensus_queue.cc:260] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.313894 16214 raft_consensus.cc:729] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8405079a669489d90f494f0fbd3bac0, State: Initialized, Role: FOLLOWER
I20260924 19:34:19.313993 16214 consensus_queue.cc:260] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.314126 15931 heartbeater.cc:503] Master 127.0.98.62:40655 was elected leader, sending a full tablet report...
I20260924 19:34:19.314114 16215 ts_tablet_manager.cc:1428] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:34:19.314337 16214 ts_tablet_manager.cc:1428] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:34:19.314354 16065 heartbeater.cc:503] Master 127.0.98.62:40655 was elected leader, sending a full tablet report...
W20260924 19:34:19.380710 16066 tablet.cc:2369] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:34:19.479766 15932 tablet.cc:2369] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:34:19.526970 16200 tablet.cc:2369] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:34:19.616812 16219 raft_consensus.cc:482] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:19.616945 16219 raft_consensus.cc:504] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.617318 16219 leader_election.cc:288] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601), f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
I20260924 19:34:19.620996 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.617590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51563 (local address 127.0.98.2:39579)
0924 19:34:19.617633 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.617634 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.617645 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:34:19.617675 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.617676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.617689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.617746 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:19.617910 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.618646 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.620030 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.620600 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.620701 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.620712 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.620882 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.620921 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:34:19.620996 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.617546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42255 (local address 127.0.98.1:33601)
0924 19:34:19.617704 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.617706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.617800 (+ 94us) server_negotiation.cc:407] Connection header received
0924 19:34:19.617852 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.617853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.617862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.617906 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:19.618022 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.618703 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.620029 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.620600 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.620687 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.620700 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.620883 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.620921 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260924 19:34:19.621022 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.617500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39579 (local address 127.0.98.3:51563)
0924 19:34:19.617536 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.617547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:19.617588 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:19.617770 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:19.617771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:19.617773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:19.617843 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.617846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.618730 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.618731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.619939 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.619942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.620704 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.620705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.620801 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.620807 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:34:19.620862 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:34:19.620923 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":7}
I20260924 19:34:19.621027 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.617500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33601 (local address 127.0.98.3:42255)
0924 19:34:19.617751 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.617759 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:19.617805 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:19.617915 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:19.617915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:19.617917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:19.617968 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.617969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.618773 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.618773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.619939 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:19.619942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.620704 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.620705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:19.620801 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.620807 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:34:19.620862 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 19:34:19.620923 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":70,"thread_start_us":95,"threads_started":1}
I20260924 19:34:19.621474 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d28c2779a7814145b43a31803fd8fe31" candidate_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95db9885af04cc5a5dfa61fc10e6804" is_pre_election: true
I20260924 19:34:19.621536 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d28c2779a7814145b43a31803fd8fe31" candidate_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8405079a669489d90f494f0fbd3bac0" is_pre_election: true
I20260924 19:34:19.621652 15885 raft_consensus.cc:2454] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc42a386b1ef48c1a29e09f81d238b7b in term 0.
I20260924 19:34:19.621665 16019 raft_consensus.cc:2454] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc42a386b1ef48c1a29e09f81d238b7b in term 0.
I20260924 19:34:19.621881 16088 leader_election.cc:302] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8405079a669489d90f494f0fbd3bac0, fc42a386b1ef48c1a29e09f81d238b7b; no voters:
I20260924 19:34:19.621996 16219 raft_consensus.cc:2781] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:34:19.622058 16219 raft_consensus.cc:482] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:34:19.622074 16219 raft_consensus.cc:3037] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:19.622751 16219 raft_consensus.cc:504] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.622865 16219 leader_election.cc:288] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [CANDIDATE]: Term 1 election: Requested vote from peers a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601), f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
I20260924 19:34:19.623011 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d28c2779a7814145b43a31803fd8fe31" candidate_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8405079a669489d90f494f0fbd3bac0"
I20260924 19:34:19.623011 15885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d28c2779a7814145b43a31803fd8fe31" candidate_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95db9885af04cc5a5dfa61fc10e6804"
I20260924 19:34:19.623082 16019 raft_consensus.cc:3037] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:19.623083 15885 raft_consensus.cc:3037] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:19.623742 15885 raft_consensus.cc:2454] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc42a386b1ef48c1a29e09f81d238b7b in term 1.
I20260924 19:34:19.623772 16019 raft_consensus.cc:2454] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc42a386b1ef48c1a29e09f81d238b7b in term 1.
I20260924 19:34:19.623898 16088 leader_election.cc:302] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8405079a669489d90f494f0fbd3bac0, fc42a386b1ef48c1a29e09f81d238b7b; no voters:
I20260924 19:34:19.623996 16219 raft_consensus.cc:2781] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:34:19.624125 16219 raft_consensus.cc:686] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 LEADER]: Becoming Leader. State: Replica: fc42a386b1ef48c1a29e09f81d238b7b, State: Running, Role: LEADER
I20260924 19:34:19.624213 16219 consensus_queue.cc:237] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:19.624989 15740 catalog_manager.cc:5495] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } health_report { overall_health: UNKNOWN } } }
I20260924 19:34:19.658892 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.656802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.0.98.3:33313)
0924 19:34:19.656846 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:19.656848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:19.656857 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:34:19.656896 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:19.656897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:19.656915 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:19.656977 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:34:19.657092 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.657552 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.658258 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:19.658439 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:19.658483 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:19.658621 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:34:19.658727 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:34:19.658783 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:19.658813 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 19:34:19.658831 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:34:19.660578 16019 raft_consensus.cc:1261] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Refusing update from remote peer fc42a386b1ef48c1a29e09f81d238b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:34:19.660605 15885 raft_consensus.cc:1261] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Refusing update from remote peer fc42a386b1ef48c1a29e09f81d238b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:34:19.660928 16219 consensus_queue.cc:1035] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:34:19.661064 16223 consensus_queue.cc:1035] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:34:19.663201 16226 mvcc.cc:205] Tried to move back new op lower bound from 7332980570767196160 to 7332980570621198336. Current Snapshot: MvccSnapshot[applied={T|T < 7332980570767196160}]
I20260924 19:34:19.663398 16228 mvcc.cc:205] Tried to move back new op lower bound from 7332980570767196160 to 7332980570621198336. Current Snapshot: MvccSnapshot[applied={T|T < 7332980570767196160}]
I20260924 19:34:19.664008 16227 mvcc.cc:205] Tried to move back new op lower bound from 7332980570767196160 to 7332980570621198336. Current Snapshot: MvccSnapshot[applied={T|T < 7332980570767196160}]
I20260924 19:34:19.665268 15739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:34:19.665346 15739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:34:19.666029 15739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Sending DeleteTablet for 3 replicas of tablet d28c2779a7814145b43a31803fd8fe31
I20260924 19:34:19.666299 16133 tablet_service.cc:1514] Processing DeleteTablet for tablet d28c2779a7814145b43a31803fd8fe31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60456
I20260924 19:34:19.666317 15865 tablet_service.cc:1514] Processing DeleteTablet for tablet d28c2779a7814145b43a31803fd8fe31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35918
I20260924 19:34:19.666360 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet d28c2779a7814145b43a31803fd8fe31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40614
I20260924 19:34:19.666549 16236 tablet_replica.cc:330] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: stopping tablet replica
I20260924 19:34:19.666574 16237 tablet_replica.cc:330] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: stopping tablet replica
I20260924 19:34:19.666608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 15802
I20260924 19:34:19.666752 16237 raft_consensus.cc:2229] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:19.668692 16236 raft_consensus.cc:2229] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:34:19.668817 16236 raft_consensus.cc:2258] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:19.669232 16236 ts_tablet_manager.cc:1905] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:19.670858 16236 ts_tablet_manager.cc:1918] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:19.670955 16236 log.cc:1196] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/d28c2779a7814145b43a31803fd8fe31
W20260924 19:34:19.670955 15727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33601 (error 108)
I20260924 19:34:19.671212 16238 tablet_replica.cc:330] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: stopping tablet replica
I20260924 19:34:19.671224 16236 ts_tablet_manager.cc:1939] T d28c2779a7814145b43a31803fd8fe31 P fc42a386b1ef48c1a29e09f81d238b7b: Deleting consensus metadata
I20260924 19:34:19.671406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 15936
I20260924 19:34:19.671547 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.671401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33601 (local address 127.0.0.1:35928)
0924 19:34:19.671466 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.671515 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 19:34:19.672288 15726 catalog_manager.cc:4841] TS fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313): tablet d28c2779a7814145b43a31803fd8fe31 (table test-table [id=1719137a2d5140b09e6f535f77d47ca0]) successfully deleted
I20260924 19:34:19.675436 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:19.675324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39579 (local address 127.0.0.1:40618)
0924 19:34:19.675380 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:19.675406 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 19:34:19.675619 15727 catalog_manager.cc:4584] TS a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d28c2779a7814145b43a31803fd8fe31: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33601: connect: Connection refused (error 111)
I20260924 19:34:19.675781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 16070
I20260924 19:34:19.679639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 15708
I20260924 19:34:39.684158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40655
--webserver_interface=127.0.98.62
--webserver_port=45933
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:34:39.783591 16240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:39.783845 16240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:39.783874 16240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:39.785292 16240 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:34:39.785341 16240 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:34:39.785352 16240 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:34:39.785363 16240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:34:39.785375 16240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:39.785387 16240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:39.785403 16240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:34:39.785430 16240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:34:39.787091 16240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:39.787379 16240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:39.787647 16240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:39.797580 16249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:39.797582 16246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:39.797580 16247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:39.797935 16240 server_base.cc:1034] running on GCE node
I20260924 19:34:39.798264 16240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:39.798631 16240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:39.799836 16240 hybrid_clock.cc:648] HybridClock initialized: now 1790278479799768 us; error 50 us; skew 500 ppm
Sep 24 19:34:39 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:39.806591 16240 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:34:39.807140 16240 webserver.cc:460] Webserver started at http://127.0.98.62:45933/ using document root <none> and password file <none>
I20260924 19:34:39.807389 16240 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:39.807454 16240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:39.809048 16240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:34:39.809746 16256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:39.809954 16240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:34:39.810053 16240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "bbdb8780f9a8456884d843009e67c5c3"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "7f3a508358fa78680d41dc0201bf5916"
server_key_iv: "3068d9c48edfd045bca05f56a8949a5e"
server_key_version: "encryptionkey@0"
I20260924 19:34:39.810504 16240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:39.825119 16240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:39.827220 16240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:39.827358 16240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:39.831544 16240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40655
I20260924 19:34:39.831610 16308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40655 every 8 connection(s)
I20260924 19:34:39.832000 16240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:34:39.833848 16309 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:34:39.834668 16309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Bootstrap starting.
I20260924 19:34:39.840291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 16240
I20260924 19:34:39.840598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33601
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40655
--webserver_port=38651
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:34:39.845266 16309 log.cc:824] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:39.847819 16309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:34:39.848153 16309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Bootstrap complete.
I20260924 19:34:39.850749 16309 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:34:39.851037 16309 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdb8780f9a8456884d843009e67c5c3, State: Initialized, Role: FOLLOWER
I20260924 19:34:39.851217 16309 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:34:39.851298 16309 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:34:39.851356 16309 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:34:39.851406 16309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:34:39.852667 16309 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:34:39.852789 16309 leader_election.cc:302] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbdb8780f9a8456884d843009e67c5c3; no voters:
I20260924 19:34:39.853014 16309 leader_election.cc:288] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:34:39.853062 16313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:34:39.853197 16313 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 2 LEADER]: Becoming Leader. State: Replica: bbdb8780f9a8456884d843009e67c5c3, State: Running, Role: LEADER
I20260924 19:34:39.853346 16309 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:34:39.853504 16313 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:34:39.853896 16314 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bbdb8780f9a8456884d843009e67c5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } } }
I20260924 19:34:39.853999 16314 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:39.854434 16315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbdb8780f9a8456884d843009e67c5c3. Latest consensus state: current_term: 2 leader_uuid: "bbdb8780f9a8456884d843009e67c5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } } }
I20260924 19:34:39.854635 16315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:34:39.854972 16321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:34:39.855993 16321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:34:39.856104 16321 catalog_manager.cc:1259] Loaded cluster ID: 0c8cddbf8efa49a5ac0f25f59790144f
I20260924 19:34:39.856138 16321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:34:39.857107 16321 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:34:39.857299 16321 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Loaded TSK: 0
I20260924 19:34:39.866813 16321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Generated new TSK 1
I20260924 19:34:39.866935 16321 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:34:39.946183 16311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:39.946379 16311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:39.946403 16311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:39.947561 16311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:39.947602 16311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:39.947628 16311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:34:39.949111 16311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38651
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:39.949313 16311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:39.949532 16311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:39.954299 16339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:39.954320 16337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:39.954403 16336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:39.954910 16311 server_base.cc:1034] running on GCE node
I20260924 19:34:39.955101 16311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:39.955325 16311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:39.956482 16311 hybrid_clock.cc:648] HybridClock initialized: now 1790278479956469 us; error 33 us; skew 500 ppm
Sep 24 19:34:39 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:39.958920 16311 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:34:39.959347 16311 webserver.cc:460] Webserver started at http://127.0.98.1:38651/ using document root <none> and password file <none>
I20260924 19:34:39.959558 16311 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:39.959618 16311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:39.961036 16311 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:34:39.961782 16346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:39.961963 16311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:34:39.962054 16311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "a95db9885af04cc5a5dfa61fc10e6804"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "1618d3909acc7ff7a8e8c8f8db381b7b"
server_key_iv: "050adb6c3f4483bf78213d992cb4fd1f"
server_key_version: "encryptionkey@0"
I20260924 19:34:39.962446 16311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:39.971755 16311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:39.973891 16311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:39.974033 16311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:39.974249 16311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:39.974661 16353 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:34:39.976531 16311 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:34:39.976583 16311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:34:39.976619 16311 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:34:39.977283 16311 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:34:39.977315 16311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:39.977406 16353 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Bootstrap starting.
I20260924 19:34:39.983843 16311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33601
I20260924 19:34:39.983986 16460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33601 every 8 connection(s)
I20260924 19:34:39.984282 16311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:34:39.986418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 16311
I20260924 19:34:39.986677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39579
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40655
--webserver_port=45261
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:34:39.988443 16353 log.cc:824] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:39.993822 16353 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:34:39.994594 16353 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Bootstrap complete.
Sep 24 19:34:39 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:39.995765 16353 ts_tablet_manager.cc:1397] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.012s
I20260924 19:34:39.997795 16353 raft_consensus.cc:348] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:39.998211 16353 raft_consensus.cc:729] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95db9885af04cc5a5dfa61fc10e6804, State: Initialized, Role: FOLLOWER
I20260924 19:34:39.998427 16353 consensus_queue.cc:260] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:39.998865 16353 ts_tablet_manager.cc:1428] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 19:34:40.000102 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:39.984987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:33067)
0924 19:34:39.985177 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:39.985230 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:39.985702 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:39.989205 (+ 3503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:39.989211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:39.989559 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:39.989824 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:39.989833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:39.991530 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:39.991532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:39.992867 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:39.992871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:39.993735 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:39.993737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:39.993884 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:39.994456 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:39.994465 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:39.996183 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:39.998317 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:39.998324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:39.998326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:39.998595 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:39.998867 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:39.998868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:39.998868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:39.999016 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:39.999453 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:39.999457 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:39.999614 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 19:34:39.999730 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:34:39.999901 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260924 19:34:40.000203 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:39.985109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33067 (local address 127.0.98.62:40655)
0924 19:34:39.988767 (+ 3658us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:39.988777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:39.988828 (+ 51us) server_negotiation.cc:407] Connection header received
0924 19:34:39.988947 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:39.988952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:39.989002 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:39.989120 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:39.989976 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:39.991393 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:39.992971 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:39.993533 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:39.993627 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:39.996309 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:39.996313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:39.996313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:39.996326 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:39.998224 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:39.998678 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:39.998681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:39.998681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:39.998790 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:39.999081 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:39.999081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:39.999082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:39.999371 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:39.999451 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:39.999787 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 19:34:39.999929 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3599,"thread_start_us":98,"threads_started":1}
I20260924 19:34:40.001180 16461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40655
I20260924 19:34:40.001283 16461 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:40.001461 16461 heartbeater.cc:511] Master 127.0.98.62:40655 requested a full tablet report, sending...
I20260924 19:34:40.002209 16273 ts_manager.cc:194] Registered new tserver with Master: a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601)
I20260924 19:34:40.003414 16273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33067
I20260924 19:34:40.004420 16461 heartbeater.cc:503] Master 127.0.98.62:40655 was elected leader, sending a full tablet report...
I20260924 19:34:40.014386 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.003060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.0.98.1:33601)
0924 19:34:40.010533 (+ 7473us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.010537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.010547 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:34:40.010584 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.010585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.010601 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.010677 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:40.010890 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.011940 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.013477 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.014002 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.014063 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.014102 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.014279 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.014317 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7434,"thread_start_us":67,"threads_started":1}
I20260924 19:34:40.014401 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.002824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33601 (local address 127.0.0.1:35626)
0924 19:34:40.010254 (+ 7430us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:40.010298 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:40.010391 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:40.010721 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:40.010724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:40.010727 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:40.010806 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.010809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.012034 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.012035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.013365 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.013369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.014083 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.014084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.014172 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.014186 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:34:40.014258 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:34:40.014320 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7387,"thread_start_us":88,"threads_started":1}
I20260924 19:34:40.014927 16395 tablet_service.cc:1514] Processing DeleteTablet for tablet d28c2779a7814145b43a31803fd8fe31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35626
I20260924 19:34:40.015488 16470 tablet_replica.cc:330] stopping tablet replica
I20260924 19:34:40.015604 16470 raft_consensus.cc:2229] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:40.015669 16470 raft_consensus.cc:2258] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:40.016010 16470 ts_tablet_manager.cc:1905] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:40.018630 16470 ts_tablet_manager.cc:1918] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:40.018704 16470 log.cc:1196] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal/wals/d28c2779a7814145b43a31803fd8fe31
I20260924 19:34:40.018921 16470 ts_tablet_manager.cc:1939] T d28c2779a7814145b43a31803fd8fe31 P a95db9885af04cc5a5dfa61fc10e6804: Deleting consensus metadata
I20260924 19:34:40.019472 16260 catalog_manager.cc:4841] TS a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601): tablet d28c2779a7814145b43a31803fd8fe31 (table test-table [id=1719137a2d5140b09e6f535f77d47ca0]) successfully deleted
W20260924 19:34:40.072300 16465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:40.072499 16465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:40.072522 16465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:40.073657 16465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:40.073695 16465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:40.073720 16465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:34:40.075114 16465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45261
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:40.075299 16465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:40.075477 16465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:40.080159 16478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:40.080163 16475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:40.080278 16476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:40.080323 16465 server_base.cc:1034] running on GCE node
I20260924 19:34:40.080586 16465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:40.080865 16465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:40.082011 16465 hybrid_clock.cc:648] HybridClock initialized: now 1790278480081994 us; error 35 us; skew 500 ppm
Sep 24 19:34:40 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:40.084364 16465 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:34:40.084892 16465 webserver.cc:460] Webserver started at http://127.0.98.2:45261/ using document root <none> and password file <none>
I20260924 19:34:40.085112 16465 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:40.085176 16465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:40.086545 16465 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:34:40.087251 16485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:40.087471 16465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:34:40.087550 16465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "f8405079a669489d90f494f0fbd3bac0"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "4f643a7248df7fcde2aed66a0a864afd"
server_key_iv: "9be175050e98cd6e0eacd84caaf8b19d"
server_key_version: "encryptionkey@0"
I20260924 19:34:40.087932 16465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:40.096555 16465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:40.098295 16465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:40.098424 16465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:40.098625 16465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:40.099006 16492 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:34:40.100453 16465 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:34:40.100504 16465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:34:40.100541 16465 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:34:40.101177 16465 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:34:40.101208 16465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:40.101303 16492 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Bootstrap starting.
I20260924 19:34:40.107292 16465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39579
I20260924 19:34:40.107385 16599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39579 every 8 connection(s)
I20260924 19:34:40.107677 16465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:34:40.110342 16492 log.cc:824] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:40.111611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 16465
I20260924 19:34:40.111878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33313
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40655
--webserver_port=42615
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:34:40.112843 16492 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:34:40.113121 16492 tablet_bootstrap.cc:492] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Bootstrap complete.
Sep 24 19:34:40 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:40.114189 16492 ts_tablet_manager.cc:1397] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260924 19:34:40.115469 16492 raft_consensus.cc:348] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:40.115796 16492 raft_consensus.cc:729] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8405079a669489d90f494f0fbd3bac0, State: Initialized, Role: FOLLOWER
I20260924 19:34:40.115971 16492 consensus_queue.cc:260] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } }
I20260924 19:34:40.116256 16492 ts_tablet_manager.cc:1428] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:34:40.117002 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.108401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45803 (local address 127.0.98.62:40655)
0924 19:34:40.108457 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.108459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.108694 (+ 235us) server_negotiation.cc:407] Connection header received
0924 19:34:40.109200 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.109202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.109216 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.109265 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:40.109849 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.110346 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.111421 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.111629 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.111683 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.113914 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:40.113915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:40.113915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:40.113922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:40.115467 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:40.115945 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:40.115946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:40.115946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:40.116011 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:40.116328 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:40.116329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:40.116329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:40.116459 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:40.116515 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.116856 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.116938 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:34:40.117276 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.108376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:45803)
0924 19:34:40.108569 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:40.108605 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:40.109102 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:40.109321 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:40.109325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:40.109573 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:40.109752 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.109757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.110445 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.110446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.111340 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.111343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.111711 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.111712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.111833 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.112305 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:40.112313 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:40.113826 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:40.115558 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:40.115565 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:40.115567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:40.115875 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:40.116114 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:40.116115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:40.116115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:40.116253 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:40.116522 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:40.116528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:40.116719 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 19:34:40.116834 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 19:34:40.117008 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":110,"threads_started":1}
I20260924 19:34:40.117748 16600 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40655
I20260924 19:34:40.117846 16600 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:40.118028 16600 heartbeater.cc:511] Master 127.0.98.62:40655 requested a full tablet report, sending...
I20260924 19:34:40.118660 16273 ts_manager.cc:194] Registered new tserver with Master: f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
I20260924 19:34:40.119163 16273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45803
I20260924 19:34:40.120005 16600 heartbeater.cc:503] Master 127.0.98.62:40655 was elected leader, sending a full tablet report...
I20260924 19:34:40.123895 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.118956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39579 (local address 127.0.0.1:35284)
0924 19:34:40.119012 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:40.119024 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:40.119068 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:40.120907 (+ 1839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:40.120908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:40.120910 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:40.120949 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.120950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.122151 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.122152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.122849 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.122851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.123710 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.123711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.123767 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.123770 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:34:40.123804 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 19:34:40.123838 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 19:34:40.123920 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.119073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.0.98.2:39579)
0924 19:34:40.120736 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.120741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.120752 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:34:40.120787 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.120789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.120805 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.120888 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:40.121001 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.122062 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.122926 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.123643 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.123715 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.123769 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.123816 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.123851 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":69,"threads_started":1}
I20260924 19:34:40.124302 16534 tablet_service.cc:1514] Processing DeleteTablet for tablet d28c2779a7814145b43a31803fd8fe31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35284
I20260924 19:34:40.124823 16607 tablet_replica.cc:330] stopping tablet replica
I20260924 19:34:40.124933 16607 raft_consensus.cc:2229] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:40.125005 16607 raft_consensus.cc:2258] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:40.125319 16607 ts_tablet_manager.cc:1905] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:40.127707 16607 ts_tablet_manager.cc:1918] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:40.127776 16607 log.cc:1196] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/d28c2779a7814145b43a31803fd8fe31
I20260924 19:34:40.128008 16607 ts_tablet_manager.cc:1939] T d28c2779a7814145b43a31803fd8fe31 P f8405079a669489d90f494f0fbd3bac0: Deleting consensus metadata
I20260924 19:34:40.128517 16259 catalog_manager.cc:4841] TS f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579): tablet d28c2779a7814145b43a31803fd8fe31 (table test-table [id=1719137a2d5140b09e6f535f77d47ca0]) successfully deleted
W20260924 19:34:40.195482 16604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:34:40.195690 16604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:34:40.195782 16604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:34:40.197003 16604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:34:40.197047 16604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:34:40.197084 16604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:34:40.198580 16604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42615
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:34:40.198827 16604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:34:40.199041 16604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:34:40.203979 16615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:40.204056 16604 server_base.cc:1034] running on GCE node
W20260924 19:34:40.203996 16612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:34:40.203979 16613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:34:40.204443 16604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:34:40.204741 16604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:34:40.205947 16604 hybrid_clock.cc:648] HybridClock initialized: now 1790278480205932 us; error 25 us; skew 500 ppm
Sep 24 19:34:40 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:34:40.208478 16604 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:34:40.208950 16604 webserver.cc:460] Webserver started at http://127.0.98.3:42615/ using document root <none> and password file <none>
I20260924 19:34:40.209146 16604 fs_manager.cc:359] Metadata directory not provided
I20260924 19:34:40.209204 16604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:34:40.210510 16604 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:34:40.211057 16622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:40.211213 16604 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:34:40.211292 16604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "fc42a386b1ef48c1a29e09f81d238b7b"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "92dcfa653bd644c30979b3769587c1da"
server_key_iv: "87ddabbf8c6f99035dbe3bed744ccab6"
server_key_version: "encryptionkey@0"
I20260924 19:34:40.211624 16604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:34:40.237629 16604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:34:40.239588 16604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:34:40.239765 16604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:34:40.240029 16604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:34:40.240329 16604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:34:40.240360 16604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:40.240381 16604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:34:40.240404 16604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:40.245697 16604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33313
I20260924 19:34:40.245736 16735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33313 every 8 connection(s)
I20260924 19:34:40.246057 16604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:34:40.247826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 16604
Sep 24 19:34:40 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:34:40.254763 16740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.248179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.0.98.62:40655)
0924 19:34:40.248550 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.248552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.248558 (+ 6us) server_negotiation.cc:407] Connection header received
0924 19:34:40.248573 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.248574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.248583 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.248667 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:40.249205 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.249898 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.250621 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.250878 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.250938 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.252073 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:40.252074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:40.252075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:40.252080 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:40.253597 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:40.253986 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:40.253987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:40.253987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:40.254040 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:40.254231 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:40.254232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:40.254232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:40.254362 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:40.254552 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.254591 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.254665 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":63,"threads_started":1}
I20260924 19:34:40.255599 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.246708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57913 (local address 127.0.98.62:40655)
0924 19:34:40.246768 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.246770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.246982 (+ 212us) server_negotiation.cc:407] Connection header received
0924 19:34:40.247475 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.247476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.247488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.247538 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:40.248518 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.249079 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.250308 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.250499 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.250551 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.252909 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:40.252911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:40.252911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:40.252917 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:40.254221 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:40.254660 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:40.254661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:40.254662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:40.254716 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:40.255011 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:40.255012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:40.255013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:40.255116 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:40.255168 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.255484 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.255544 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:34:40.255833 16738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.246704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:57913)
0924 19:34:40.246870 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:40.246906 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:40.247377 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:40.247606 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:40.247610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:40.248212 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:34:40.248433 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.248437 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.249160 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.249161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.250236 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.250239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.250565 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.250566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.250701 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.251286 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:34:40.251294 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:34:40.252838 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:34:40.254284 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:40.254291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:40.254293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:40.254594 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:40.254778 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:34:40.254779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:34:40.254779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:34:40.254913 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:34:40.255176 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:34:40.255181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:34:40.255345 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 19:34:40.255483 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 19:34:40.255620 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260924 19:34:40.256196 16736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40655
I20260924 19:34:40.256278 16736 heartbeater.cc:464] Registering TS with master...
I20260924 19:34:40.256419 16736 heartbeater.cc:511] Master 127.0.98.62:40655 requested a full tablet report, sending...
I20260924 19:34:40.256747 16273 ts_manager.cc:194] Registered new tserver with Master: fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313)
I20260924 19:34:40.257062 16273 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57913
I20260924 19:34:40.266417 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:34:40.269554 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.266818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.0.98.62:40655)
0924 19:34:40.266911 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.266913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.267009 (+ 96us) server_negotiation.cc:407] Connection header received
0924 19:34:40.267088 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.267089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.267127 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.267186 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:34:40.267328 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.267836 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.268556 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.268830 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.268964 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.269262 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:34:40.269321 (+ 59us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 19:34:40.269488 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48102: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 19:34:40.269613 16464 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48102: token expired
W20260924 19:34:40.269688 16204 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:40655)
I20260924 19:34:40.275275 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.270005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.0.98.62:40655)
0924 19:34:40.270052 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.270054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.270061 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:34:40.270080 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.270081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.270089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.270131 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:34:40.270567 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.271007 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.271708 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.271907 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.271955 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.273062 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:34:40.273063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:34:40.273063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:34:40.273069 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:34:40.274280 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:40.274580 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:40.274581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:40.274581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:40.274633 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:34:40.274789 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:34:40.274789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:34:40.274790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:34:40.274933 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:34:40.275019 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.275116 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.275190 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:34:40.276440 16273 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:34:40.278087 16273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:34:40.279955 16534 tablet_service.cc:1467] Processing CreateTablet for tablet 549696b854f8417da260e52f24176da3 (DEFAULT_TABLE table=test-table [id=6c8e4b31609246768798d4b24f1b75d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:40.280076 16395 tablet_service.cc:1467] Processing CreateTablet for tablet 549696b854f8417da260e52f24176da3 (DEFAULT_TABLE table=test-table [id=6c8e4b31609246768798d4b24f1b75d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:40.280126 16534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549696b854f8417da260e52f24176da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:40.280210 16395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549696b854f8417da260e52f24176da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:40.281366 16353 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Bootstrap starting.
I20260924 19:34:40.281574 16492 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: Bootstrap starting.
I20260924 19:34:40.281680 16353 tablet_bootstrap.cc:654] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:40.281843 16492 tablet_bootstrap.cc:654] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:40.282382 16492 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: No bootstrap required, opened a new log
I20260924 19:34:40.282399 16353 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: No bootstrap required, opened a new log
I20260924 19:34:40.282433 16492 ts_tablet_manager.cc:1397] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:40.282434 16353 ts_tablet_manager.cc:1397] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 19:34:40.282528 16353 raft_consensus.cc:348] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.282529 16492 raft_consensus.cc:348] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.282596 16492 raft_consensus.cc:374] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:40.282595 16353 raft_consensus.cc:374] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:40.282645 16492 raft_consensus.cc:729] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8405079a669489d90f494f0fbd3bac0, State: Initialized, Role: FOLLOWER
I20260924 19:34:40.282645 16353 raft_consensus.cc:729] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95db9885af04cc5a5dfa61fc10e6804, State: Initialized, Role: FOLLOWER
I20260924 19:34:40.282687 16492 consensus_queue.cc:260] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.282688 16353 consensus_queue.cc:260] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.282774 16492 ts_tablet_manager.cc:1428] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:40.282775 16353 ts_tablet_manager.cc:1428] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 19:34:40.283555 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.279695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33313 (local address 127.0.0.1:55786)
0924 19:34:40.279825 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:40.279837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:40.279879 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:40.280259 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:40.280260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:40.280264 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:40.280315 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.280317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.281553 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.281554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.282497 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.282501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.283376 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.283377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.283428 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.283431 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:34:40.283475 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:34:40.283507 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 19:34:40.283576 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.279880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.0.98.3:33313)
0924 19:34:40.280090 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.280095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.280106 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:34:40.280145 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.280147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.280163 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.280248 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:40.280379 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.281451 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.282593 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.283271 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.283374 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.283420 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.283487 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.283516 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":119,"threads_started":1}
I20260924 19:34:40.284227 16670 tablet_service.cc:1467] Processing CreateTablet for tablet 549696b854f8417da260e52f24176da3 (DEFAULT_TABLE table=test-table [id=6c8e4b31609246768798d4b24f1b75d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:34:40.284520 16670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549696b854f8417da260e52f24176da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:34:40.287146 16745 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: Bootstrap starting.
I20260924 19:34:40.287678 16745 tablet_bootstrap.cc:654] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: Neither blocks nor log segments found. Creating new log.
I20260924 19:34:40.288144 16745 log.cc:824] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:34:40.288798 16745 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: No bootstrap required, opened a new log
I20260924 19:34:40.288852 16745 ts_tablet_manager.cc:1397] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:34:40.290267 16745 raft_consensus.cc:348] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.290436 16745 raft_consensus.cc:374] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:34:40.290458 16745 raft_consensus.cc:729] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc42a386b1ef48c1a29e09f81d238b7b, State: Initialized, Role: FOLLOWER
I20260924 19:34:40.290586 16745 consensus_queue.cc:260] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.290815 16745 ts_tablet_manager.cc:1428] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:34:40.290831 16736 heartbeater.cc:503] Master 127.0.98.62:40655 was elected leader, sending a full tablet report...
I20260924 19:34:40.313074 16605 raft_consensus.cc:482] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:34:40.313217 16605 raft_consensus.cc:504] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.313555 16605 leader_election.cc:288] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313), a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601)
I20260924 19:34:40.316556 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.313716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33601 (local address 127.0.98.2:36399)
0924 19:34:40.313773 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:40.313783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:40.313822 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:40.313952 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:40.313953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:40.313955 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:40.314013 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.314015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.314741 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.314743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.315697 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.315700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.316316 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.316317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.316374 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.316377 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:34:40.316424 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:34:40.316464 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:34:40.316565 16469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.313778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36399 (local address 127.0.98.1:33601)
0924 19:34:40.313833 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.313835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.313845 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:34:40.313875 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.313876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.313888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.313946 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:40.314078 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.314679 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.315802 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.316247 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.316310 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.316320 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.316440 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.316489 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 19:34:40.316962 16415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549696b854f8417da260e52f24176da3" candidate_uuid: "f8405079a669489d90f494f0fbd3bac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95db9885af04cc5a5dfa61fc10e6804" is_pre_election: true
I20260924 19:34:40.317065 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.313737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33313 (local address 127.0.98.2:55935)
0924 19:34:40.313943 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:40.313953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:34:40.314003 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:34:40.314123 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:34:40.314124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:34:40.314126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:34:40.314215 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.314216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.315129 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.315129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.316007 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:34:40.316010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.316829 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.316830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:34:40.316905 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.316909 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:34:40.316962 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:34:40.316994 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":141,"threads_started":1}
I20260924 19:34:40.317081 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.313906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55935 (local address 127.0.98.3:33313)
0924 19:34:40.313966 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.313968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.313992 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:34:40.314057 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.314058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.314069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.314115 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:34:40.314333 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.315044 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.316092 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.316581 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.316677 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.316689 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.316980 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.317023 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:34:40.317139 16415 raft_consensus.cc:2454] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8405079a669489d90f494f0fbd3bac0 in term 0.
I20260924 19:34:40.317313 16690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549696b854f8417da260e52f24176da3" candidate_uuid: "f8405079a669489d90f494f0fbd3bac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" is_pre_election: true
I20260924 19:34:40.317373 16489 leader_election.cc:302] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a95db9885af04cc5a5dfa61fc10e6804, f8405079a669489d90f494f0fbd3bac0; no voters:
I20260924 19:34:40.317430 16690 raft_consensus.cc:2454] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8405079a669489d90f494f0fbd3bac0 in term 0.
I20260924 19:34:40.317497 16605 raft_consensus.cc:2781] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:34:40.317560 16605 raft_consensus.cc:482] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:34:40.317593 16605 raft_consensus.cc:3037] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:40.318320 16605 raft_consensus.cc:504] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.318456 16605 leader_election.cc:288] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [CANDIDATE]: Term 1 election: Requested vote from peers fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313), a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601)
I20260924 19:34:40.318611 16690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549696b854f8417da260e52f24176da3" candidate_uuid: "f8405079a669489d90f494f0fbd3bac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc42a386b1ef48c1a29e09f81d238b7b"
I20260924 19:34:40.318656 16415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549696b854f8417da260e52f24176da3" candidate_uuid: "f8405079a669489d90f494f0fbd3bac0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a95db9885af04cc5a5dfa61fc10e6804"
I20260924 19:34:40.318704 16690 raft_consensus.cc:3037] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:40.318791 16415 raft_consensus.cc:3037] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:34:40.319491 16690 raft_consensus.cc:2454] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8405079a669489d90f494f0fbd3bac0 in term 1.
I20260924 19:34:40.319662 16488 leader_election.cc:302] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8405079a669489d90f494f0fbd3bac0, fc42a386b1ef48c1a29e09f81d238b7b; no voters:
I20260924 19:34:40.319748 16605 raft_consensus.cc:2781] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:34:40.319849 16415 raft_consensus.cc:2454] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8405079a669489d90f494f0fbd3bac0 in term 1.
I20260924 19:34:40.319887 16605 raft_consensus.cc:686] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 1 LEADER]: Becoming Leader. State: Replica: f8405079a669489d90f494f0fbd3bac0, State: Running, Role: LEADER
I20260924 19:34:40.319952 16605 consensus_queue.cc:237] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:34:40.320752 16272 catalog_manager.cc:5495] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8405079a669489d90f494f0fbd3bac0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f8405079a669489d90f494f0fbd3bac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } health_report { overall_health: UNKNOWN } } }
I20260924 19:34:40.330452 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.328361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.0.98.2:39579)
0924 19:34:40.328402 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:34:40.328404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:34:40.328414 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:34:40.328474 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:34:40.328475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:34:40.328493 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:34:40.328550 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:34:40.328650 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.329135 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.329855 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:34:40.330051 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:34:40.330093 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:34:40.330196 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:34:40.330288 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:34:40.330333 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:34:40.330365 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 19:34:40.330382 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:34:40.331902 16690 raft_consensus.cc:1261] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 FOLLOWER]: Refusing update from remote peer f8405079a669489d90f494f0fbd3bac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:34:40.331902 16415 raft_consensus.cc:1261] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Refusing update from remote peer f8405079a669489d90f494f0fbd3bac0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:34:40.332213 16605 consensus_queue.cc:1035] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:34:40.332345 16749 consensus_queue.cc:1035] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:34:40.334314 16752 mvcc.cc:205] Tried to move back new op lower bound from 7332980655437029376 to 7332980655391129600. Current Snapshot: MvccSnapshot[applied={T|T < 7332980655437029376}]
I20260924 19:34:40.336527 16272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:34:40.336656 16272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:34:40.337352 16272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Sending DeleteTablet for 3 replicas of tablet 549696b854f8417da260e52f24176da3
I20260924 19:34:40.337625 16395 tablet_service.cc:1514] Processing DeleteTablet for tablet 549696b854f8417da260e52f24176da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35626
I20260924 19:34:40.337630 16534 tablet_service.cc:1514] Processing DeleteTablet for tablet 549696b854f8417da260e52f24176da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35284
I20260924 19:34:40.337646 16670 tablet_service.cc:1514] Processing DeleteTablet for tablet 549696b854f8417da260e52f24176da3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:34:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55786
I20260924 19:34:40.337714 16470 tablet_replica.cc:330] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: stopping tablet replica
I20260924 19:34:40.337715 16607 tablet_replica.cc:330] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: stopping tablet replica
I20260924 19:34:40.337723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 16311
I20260924 19:34:40.337759 16607 raft_consensus.cc:2229] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:34:40.337760 16470 raft_consensus.cc:2229] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:40.337818 16470 raft_consensus.cc:2258] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:40.337827 16607 raft_consensus.cc:2258] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:40.337976 16762 tablet_replica.cc:330] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: stopping tablet replica
I20260924 19:34:40.338063 16762 raft_consensus.cc:2229] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:34:40.338155 16762 raft_consensus.cc:2258] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:34:40.338582 16762 ts_tablet_manager.cc:1905] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:40.338702 16607 ts_tablet_manager.cc:1905] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:34:40.340332 16762 ts_tablet_manager.cc:1918] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:40.340401 16762 log.cc:1196] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/549696b854f8417da260e52f24176da3
I20260924 19:34:40.340401 16607 ts_tablet_manager.cc:1918] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:34:40.340464 16607 log.cc:1196] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/549696b854f8417da260e52f24176da3
I20260924 19:34:40.340682 16607 ts_tablet_manager.cc:1939] T 549696b854f8417da260e52f24176da3 P f8405079a669489d90f494f0fbd3bac0: Deleting consensus metadata
I20260924 19:34:40.340725 16762 ts_tablet_manager.cc:1939] T 549696b854f8417da260e52f24176da3 P fc42a386b1ef48c1a29e09f81d238b7b: Deleting consensus metadata
I20260924 19:34:40.341109 16259 catalog_manager.cc:4841] TS f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579): tablet 549696b854f8417da260e52f24176da3 (table test-table [id=6c8e4b31609246768798d4b24f1b75d2]) successfully deleted
I20260924 19:34:40.341192 16259 catalog_manager.cc:4841] TS fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313): tablet 549696b854f8417da260e52f24176da3 (table test-table [id=6c8e4b31609246768798d4b24f1b75d2]) successfully deleted
W20260924 19:34:40.343624 16260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33601 (error 108)
I20260924 19:34:40.343835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 16465
I20260924 19:34:40.344377 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:34:40.343971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33601 (local address 127.0.0.1:35642)
0924 19:34:40.344310 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 19:34:40.344350 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319}
W20260924 19:34:40.344508 16260 catalog_manager.cc:4584] TS a95db9885af04cc5a5dfa61fc10e6804 (127.0.98.1:33601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 549696b854f8417da260e52f24176da3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33601: connect: Connection refused (error 111)
I20260924 19:34:40.348315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 16604
I20260924 19:34:40.353048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 16240
I20260924 19:35:00.359989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40655
--webserver_interface=127.0.98.62
--webserver_port=45933
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:00.441186 16764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:00.441381 16764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:00.441402 16764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:00.442488 16764 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:35:00.442518 16764 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:35:00.442529 16764 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:35:00.442539 16764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:00.442550 16764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:00.442561 16764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:00.442572 16764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:00.442589 16764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:00.444016 16764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:00.444202 16764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:00.444371 16764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:00.449054 16764 server_base.cc:1034] running on GCE node
W20260924 19:35:00.449157 16769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:00.449160 16770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:00.449337 16772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:00.449554 16764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:00.449847 16764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:00.450982 16764 hybrid_clock.cc:648] HybridClock initialized: now 1790278500450972 us; error 32 us; skew 500 ppm
Sep 24 19:35:00 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:00.459203 16764 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:00.459689 16764 webserver.cc:460] Webserver started at http://127.0.98.62:45933/ using document root <none> and password file <none>
I20260924 19:35:00.459941 16764 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:00.459998 16764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:00.461485 16764 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:00.462172 16779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:00.462380 16764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:00.462471 16764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "bbdb8780f9a8456884d843009e67c5c3"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "7f3a508358fa78680d41dc0201bf5916"
server_key_iv: "3068d9c48edfd045bca05f56a8949a5e"
server_key_version: "encryptionkey@0"
I20260924 19:35:00.462829 16764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:00.479403 16764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:00.481442 16764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:00.481564 16764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:00.485318 16764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40655
I20260924 19:35:00.485360 16831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40655 every 8 connection(s)
I20260924 19:35:00.485700 16764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:00.487062 16832 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:35:00.487701 16832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Bootstrap starting.
I20260924 19:35:00.494762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 16764
I20260924 19:35:00.495023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33601
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40655
--webserver_port=38651
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:35:00.495803 16832 log.cc:824] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:00.497991 16832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:35:00.498275 16832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Bootstrap complete.
I20260924 19:35:00.500322 16832 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:35:00.500550 16832 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbdb8780f9a8456884d843009e67c5c3, State: Initialized, Role: FOLLOWER
I20260924 19:35:00.500748 16832 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:35:00.500829 16832 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:00.500880 16832 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:00.500926 16832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:35:00.501931 16832 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:35:00.502027 16832 leader_election.cc:302] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbdb8780f9a8456884d843009e67c5c3; no voters:
I20260924 19:35:00.502198 16832 leader_election.cc:288] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:35:00.502269 16836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:35:00.502454 16832 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:00.502472 16836 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [term 3 LEADER]: Becoming Leader. State: Replica: bbdb8780f9a8456884d843009e67c5c3, State: Running, Role: LEADER
I20260924 19:35:00.502676 16836 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } }
I20260924 19:35:00.502974 16836 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbdb8780f9a8456884d843009e67c5c3. Latest consensus state: current_term: 3 leader_uuid: "bbdb8780f9a8456884d843009e67c5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } } }
I20260924 19:35:00.503041 16836 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:00.503213 16844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:00.503471 16837 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bbdb8780f9a8456884d843009e67c5c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbdb8780f9a8456884d843009e67c5c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40655 } } }
I20260924 19:35:00.503577 16837 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:00.504177 16844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:00.504292 16844 catalog_manager.cc:1259] Loaded cluster ID: 0c8cddbf8efa49a5ac0f25f59790144f
I20260924 19:35:00.504550 16844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:00.505728 16844 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:00.505946 16844 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Loaded TSK: 0
I20260924 19:35:00.506021 16844 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbdb8780f9a8456884d843009e67c5c3: Loaded TSK: 1
F20260924 19:35:00.506276 16844 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb4a1c849ad google::LogMessage::Fail() at ??:0
@ 0x7fb4a1c8788a google::LogMessage::SendToLog() at ??:0
@ 0x7fb4a1c844de google::LogMessage::Flush() at ??:0
@ 0x7fb4a1c85a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb4a43f1104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb4a43eed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb4a9bde1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb4a9bde48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb4a9bde594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb4a9ba4c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb4a9baead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb4a3b520d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb4a3b47954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb4a49ab6db start_thread at ??:0
@ 0x7fb4a078261f clone at ??:0
W20260924 19:35:00.590049 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:00.590234 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:00.590255 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:00.591413 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:00.591459 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:00.591483 16834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:35:00.592936 16834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38651
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:00.593148 16834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:00.593369 16834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:00.598150 16834 server_base.cc:1034] running on GCE node
W20260924 19:35:00.598059 16873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:00.598057 16876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:00.598265 16874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:00.598551 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:00.598786 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:00.599929 16834 hybrid_clock.cc:648] HybridClock initialized: now 1790278500599919 us; error 36 us; skew 500 ppm
Sep 24 19:35:00 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:00.602555 16834 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:35:00.603029 16834 webserver.cc:460] Webserver started at http://127.0.98.1:38651/ using document root <none> and password file <none>
I20260924 19:35:00.603241 16834 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:00.603293 16834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:00.604826 16834 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:00.605679 16883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:00.605850 16834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:35:00.605958 16834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "a95db9885af04cc5a5dfa61fc10e6804"
format_stamp: "Formatted at 2026-09-24 19:34:18 on dist-test-slave-m8jb"
server_key: "1618d3909acc7ff7a8e8c8f8db381b7b"
server_key_iv: "050adb6c3f4483bf78213d992cb4fd1f"
server_key_version: "encryptionkey@0"
I20260924 19:35:00.606454 16834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:00.622570 16834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:00.625336 16834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:00.625495 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:00.625764 16834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:00.626341 16890 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 19:35:00.627777 16834 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:35:00.627837 16834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 19:35:00.627871 16834 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:35:00.628754 16834 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:35:00.628800 16834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:00.628898 16890 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Bootstrap starting.
I20260924 19:35:00.635807 16834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33601
I20260924 19:35:00.635928 16997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33601 every 8 connection(s)
I20260924 19:35:00.636294 16834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:35:00.639248 16890 log.cc:824] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:00.639883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 16834
I20260924 19:35:00.640167 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39579
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40655
--webserver_port=45261
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:35:00.642169 16890 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:35:00.642460 16890 tablet_bootstrap.cc:492] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Bootstrap complete.
I20260924 19:35:00.643222 16890 ts_tablet_manager.cc:1397] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260924 19:35:00.644273 16890 raft_consensus.cc:348] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:35:00.644495 16890 raft_consensus.cc:729] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a95db9885af04cc5a5dfa61fc10e6804, State: Initialized, Role: FOLLOWER
I20260924 19:35:00.644673 16890 consensus_queue.cc:260] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:35:00.644942 16890 ts_tablet_manager.cc:1428] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
W20260924 19:35:00.715567 17000 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:00.637020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:38477)
0924 19:35:00.639473 (+ 2453us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.639528 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:00.640104 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:00.715354 (+ 75250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2400,"thread_start_us":70,"threads_started":1}
W20260924 19:35:00.715896 16998 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40655 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40655: Client connection negotiation failed: client connection to 127.0.98.62:40655: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:35:00.716238 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:00.716117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:45971)
0924 19:35:00.716163 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.716220 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:35:00.716511 17000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:00.716435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:36805)
0924 19:35:00.716474 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.716495 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:35:00.716610 16998 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:35:00.735999 17002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:00.736207 17002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:00.736239 17002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:00.737493 17002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:00.737536 17002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:00.737572 17002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:35:00.739015 17002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45261
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:00.739233 17002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:00.739441 17002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:00.744202 17011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:00.744287 17008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:00.744189 17009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:00.744578 17002 server_base.cc:1034] running on GCE node
I20260924 19:35:00.744812 17002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:00.745136 17002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:00.746294 17002 hybrid_clock.cc:648] HybridClock initialized: now 1790278500746281 us; error 35 us; skew 500 ppm
Sep 24 19:35:00 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:00.748737 17002 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:35:00.749157 17002 webserver.cc:460] Webserver started at http://127.0.98.2:45261/ using document root <none> and password file <none>
I20260924 19:35:00.749347 17002 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:00.749387 17002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:00.750665 17002 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:35:00.751248 17018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:00.751400 17002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:00.751487 17002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "f8405079a669489d90f494f0fbd3bac0"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "4f643a7248df7fcde2aed66a0a864afd"
server_key_iv: "9be175050e98cd6e0eacd84caaf8b19d"
server_key_version: "encryptionkey@0"
I20260924 19:35:00.751845 17002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:00.784281 17002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:00.786574 17002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:00.786722 17002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:00.786971 17002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:00.787268 17002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:00.787304 17002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:00.787336 17002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:00.787364 17002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:00.793618 17002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39579
I20260924 19:35:00.793677 17131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39579 every 8 connection(s)
I20260924 19:35:00.794008 17002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:35:00.795043 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:00.794636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:47621)
0924 19:35:00.794809 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.794883 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
W20260924 19:35:00.795421 17132 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40655 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40655: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
I20260924 19:35:00.795691 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:00.795592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:38991)
0924 19:35:00.795639 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.795669 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:35:00.795980 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:00.795903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:60155)
0924 19:35:00.795937 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.795961 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 19:35:00.796044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 17002
W20260924 19:35:00.796111 17132 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:35:00.796290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33313
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40655
--webserver_port=42615
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:00.874393 17135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:00.874601 17135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:00.874631 17135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:00.875810 17135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:00.875854 17135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:00.875886 17135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:35:00.877351 17135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42615
--tserver_master_addrs=127.0.98.62:40655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:00.877586 17135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:00.877836 17135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:00.882613 17140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:00.882666 17141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:00.882613 17143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:00.882700 17135 server_base.cc:1034] running on GCE node
I20260924 19:35:00.883001 17135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:00.883208 17135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:00.884361 17135 hybrid_clock.cc:648] HybridClock initialized: now 1790278500884349 us; error 28 us; skew 500 ppm
Sep 24 19:35:00 dist-test-slave-m8jb krb5kdc[15695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:00.886782 17135 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:35:00.887257 17135 webserver.cc:460] Webserver started at http://127.0.98.3:42615/ using document root <none> and password file <none>
I20260924 19:35:00.887455 17135 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:00.887511 17135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:00.888922 17135 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:00.889535 17150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:00.889714 17135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:00.889801 17135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "fc42a386b1ef48c1a29e09f81d238b7b"
format_stamp: "Formatted at 2026-09-24 19:34:19 on dist-test-slave-m8jb"
server_key: "92dcfa653bd644c30979b3769587c1da"
server_key_iv: "87ddabbf8c6f99035dbe3bed744ccab6"
server_key_version: "encryptionkey@0"
I20260924 19:35:00.890137 17135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:00.906255 17135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:00.908442 17135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:00.908581 17135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:00.908831 17135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:00.909118 17135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:00.909150 17135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:00.909178 17135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:00.909196 17135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:00.914697 17135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33313
I20260924 19:35:00.914743 17263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33313 every 8 connection(s)
I20260924 19:35:00.915057 17135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:35:00.916014 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:00.915631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:46999)
0924 19:35:00.915802 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.915873 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
W20260924 19:35:00.916363 17264 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40655 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40655: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
I20260924 19:35:00.916604 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:00.916515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:33069)
0924 19:35:00.916557 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.916588 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:35:00.916837 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:00.916775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:47007)
0924 19:35:00.916804 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:00.916823 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 19:35:00.916930 17264 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:35:00.920931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 17135
I20260924 19:35:01.717242 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:01.716994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:46939)
0924 19:35:01.717128 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:01.717171 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260924 19:35:01.796538 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:01.796356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:59611)
0924 19:35:01.796450 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:01.796486 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
I20260924 19:35:01.917423 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:01.917224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:36189)
0924 19:35:01.917335 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:01.917372 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260924 19:35:02.227607 17271 raft_consensus.cc:482] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:02.227726 17271 raft_consensus.cc:504] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:35:02.228070 17271 leader_election.cc:288] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313), f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
Sep 24 19:35:02 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 19:35:02 dist-test-slave-m8jb krb5kdc[15695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 19:35:02.238283 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:02.228517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51789 (local address 127.0.98.2:39579)
0924 19:35:02.228668 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:02.228676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:02.228707 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:35:02.229236 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:02.229238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:02.229290 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:02.229441 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:02.230130 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:02.230998 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:02.231944 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:02.232350 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:02.232445 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:02.235047 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:02.235050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:02.235051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:02.235061 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:02.236674 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:02.237101 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:02.237104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:02.237105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:02.237205 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:02.237441 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:02.237442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:02.237442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:02.237699 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:02.237763 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:02.238047 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 19:35:02.238174 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":48,"threads_started":1}
I20260924 19:35:02.238276 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:02.228245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33313 (local address 127.0.98.1:54829)
0924 19:35:02.228378 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:02.228391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:02.228460 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:02.229494 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:02.229496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:02.229810 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:02.230007 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:02.230013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:02.231083 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:02.231084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:02.231865 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:02.231867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:02.232429 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:02.232430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:02.232550 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:02.233147 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:02.233152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:02.234603 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:02.236547 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:02.236556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:02.236558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:02.237010 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:02.237267 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:02.237268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:02.237268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:02.237383 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:02.237775 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:02.237776 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:02.237895 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 19:35:02.237983 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:35:02.238162 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":56,"thread_start_us":45,"threads_started":1}
I20260924 19:35:02.238286 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:02.228350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54829 (local address 127.0.98.3:33313)
0924 19:35:02.228575 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:02.228586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:02.228624 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:35:02.229228 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:02.229232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:02.229290 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:02.229441 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:02.230128 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:02.230985 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:02.231948 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:02.232346 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:02.232441 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:02.234710 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:02.234715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:02.234715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:02.234728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:02.236439 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:02.237085 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:02.237089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:02.237089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:02.237186 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:02.237445 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:02.237446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:02.237446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:02.237707 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:02.237782 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:02.238048 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 19:35:02.238177 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260924 19:35:02.238355 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:02.228380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39579 (local address 127.0.98.1:51789)
0924 19:35:02.228461 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:02.228470 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:02.228518 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:02.229562 (+ 1044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:02.229563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:02.229807 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:02.230007 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:02.230013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:02.231084 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:02.231085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:02.231864 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:02.231866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:02.232435 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:02.232436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:02.232543 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:02.233117 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:02.233125 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:02.234954 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:02.236755 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:02.236756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:02.236756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:02.237011 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:02.237266 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:02.237267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:02.237267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:02.237379 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:02.237762 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:02.237769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:02.237893 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 19:35:02.237983 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:35:02.238235 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":120,"thread_start_us":33,"threads_started":1}
W20260924 19:35:02.239063 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
W20260924 19:35:02.239233 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
I20260924 19:35:02.239281 16886 leader_election.cc:302] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a95db9885af04cc5a5dfa61fc10e6804; no voters: f8405079a669489d90f494f0fbd3bac0, fc42a386b1ef48c1a29e09f81d238b7b
I20260924 19:35:02.239408 17271 raft_consensus.cc:2726] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:35:02.717854 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:02.717707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:58665)
0924 19:35:02.717768 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:02.717812 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:35:02.797123 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:02.796924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:44657)
0924 19:35:02.797046 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:02.797083 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260924 19:35:02.917987 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:02.917805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:59697)
0924 19:35:02.917914 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:02.917950 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
I20260924 19:35:03.718436 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:03.718223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:54731)
0924 19:35:03.718343 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:03.718372 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260924 19:35:03.797734 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:03.797518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:59565)
0924 19:35:03.797653 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:03.797697 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":53,"threads_started":1}
I20260924 19:35:03.918581 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:03.918397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:36619)
0924 19:35:03.918509 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:03.918539 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260924 19:35:04.453289 17281 raft_consensus.cc:482] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:04.453413 17281 raft_consensus.cc:504] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:35:04.453614 17281 leader_election.cc:288] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313), f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
W20260924 19:35:04.453984 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
W20260924 19:35:04.454061 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
I20260924 19:35:04.454092 16886 leader_election.cc:302] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a95db9885af04cc5a5dfa61fc10e6804; no voters: f8405079a669489d90f494f0fbd3bac0, fc42a386b1ef48c1a29e09f81d238b7b
I20260924 19:35:04.454180 17281 raft_consensus.cc:2726] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:35:04.719074 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:04.718819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:46969)
0924 19:35:04.718958 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:04.719003 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260924 19:35:04.798396 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:04.798144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:57341)
0924 19:35:04.798290 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:04.798340 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260924 19:35:04.919251 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:04.919025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:57165)
0924 19:35:04.919158 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:04.919199 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260924 19:35:05.719810 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:05.719573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:46491)
0924 19:35:05.719712 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:05.719758 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260924 19:35:05.799101 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:05.798893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:49189)
0924 19:35:05.799017 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:05.799056 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260924 19:35:05.919871 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:05.919665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:47927)
0924 19:35:05.919787 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:05.919831 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260924 19:35:06.720438 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:06.720239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:55529)
0924 19:35:06.720357 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:06.720391 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260924 19:35:06.799675 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:06.799475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:56641)
0924 19:35:06.799591 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:06.799631 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260924 19:35:06.920501 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:06.920281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:37783)
0924 19:35:06.920421 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:06.920459 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260924 19:35:07.476045 17291 raft_consensus.cc:482] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:07.476155 17291 raft_consensus.cc:504] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:35:07.476311 17291 leader_election.cc:288] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313), f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
W20260924 19:35:07.476716 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
W20260924 19:35:07.476778 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
I20260924 19:35:07.476791 16886 leader_election.cc:302] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a95db9885af04cc5a5dfa61fc10e6804; no voters: f8405079a669489d90f494f0fbd3bac0, fc42a386b1ef48c1a29e09f81d238b7b
I20260924 19:35:07.476861 17291 raft_consensus.cc:2726] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:35:07.721206 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:07.720947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:50459)
0924 19:35:07.721110 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:07.721154 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260924 19:35:07.800259 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:07.800066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:38625)
0924 19:35:07.800182 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:07.800217 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260924 19:35:07.921207 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:07.920971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:58131)
0924 19:35:07.921112 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:07.921157 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260924 19:35:08.722114 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:08.721784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:34917)
0924 19:35:08.722000 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:08.722061 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":126,"threads_started":1}
I20260924 19:35:08.801050 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:08.800793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:34875)
0924 19:35:08.800963 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:08.801007 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260924 19:35:08.922042 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:08.921742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:57045)
0924 19:35:08.921913 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:08.921979 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260924 19:35:09.144127 17298 raft_consensus.cc:482] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:09.144266 17298 raft_consensus.cc:504] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:35:09.144469 17298 leader_election.cc:288] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313), f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
W20260924 19:35:09.145015 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
W20260924 19:35:09.145125 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
I20260924 19:35:09.145141 16886 leader_election.cc:302] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a95db9885af04cc5a5dfa61fc10e6804; no voters: f8405079a669489d90f494f0fbd3bac0, fc42a386b1ef48c1a29e09f81d238b7b
I20260924 19:35:09.145226 17298 raft_consensus.cc:2726] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:35:09.722754 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:09.722531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:42733)
0924 19:35:09.722673 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:09.722707 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":58,"threads_started":1}
I20260924 19:35:09.801741 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:09.801502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:60739)
0924 19:35:09.801649 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:09.801691 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260924 19:35:09.922721 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:09.922504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:34345)
0924 19:35:09.922626 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:09.922671 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260924 19:35:10.723472 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:10.723253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:49255)
0924 19:35:10.723384 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:10.723423 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260924 19:35:10.802356 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:10.802159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:35131)
0924 19:35:10.802280 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:10.802314 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260924 19:35:10.923424 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:10.923199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:39275)
0924 19:35:10.923332 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:10.923377 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260924 19:35:11.724207 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:11.723959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:37139)
0924 19:35:11.724107 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:11.724148 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260924 19:35:11.802996 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:11.802760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:55853)
0924 19:35:11.802906 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:11.802945 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260924 19:35:11.923978 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:11.923787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:38941)
0924 19:35:11.923907 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:11.923939 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260924 19:35:12.598136 17308 raft_consensus.cc:482] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:12.598228 17308 raft_consensus.cc:504] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc42a386b1ef48c1a29e09f81d238b7b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33313 } } peers { permanent_uuid: "f8405079a669489d90f494f0fbd3bac0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39579 } } peers { permanent_uuid: "a95db9885af04cc5a5dfa61fc10e6804" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33601 } }
I20260924 19:35:12.598378 17308 leader_election.cc:288] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313), f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579)
W20260924 19:35:12.598786 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f8405079a669489d90f494f0fbd3bac0 (127.0.98.2:39579): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
W20260924 19:35:12.598858 16886 leader_election.cc:341] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fc42a386b1ef48c1a29e09f81d238b7b (127.0.98.3:33313): Not found: Tablet not found: 549696b854f8417da260e52f24176da3
I20260924 19:35:12.598897 16886 leader_election.cc:302] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a95db9885af04cc5a5dfa61fc10e6804; no voters: f8405079a669489d90f494f0fbd3bac0, fc42a386b1ef48c1a29e09f81d238b7b
I20260924 19:35:12.598979 17308 raft_consensus.cc:2726] T 549696b854f8417da260e52f24176da3 P a95db9885af04cc5a5dfa61fc10e6804 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:35:12.724967 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:12.724729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:40777)
0924 19:35:12.724854 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:12.724921 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260924 19:35:12.803650 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:12.803450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:56709)
0924 19:35:12.803563 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:12.803606 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260924 19:35:12.924681 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:12.924433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:54481)
0924 19:35:12.924560 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:12.924598 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260924 19:35:13.725744 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:13.725450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:34025)
0924 19:35:13.725637 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:13.725689 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":64,"threads_started":1}
I20260924 19:35:13.804313 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:13.804082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:34523)
0924 19:35:13.804206 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:13.804243 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260924 19:35:13.925441 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:13.925192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:48617)
0924 19:35:13.925351 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:13.925393 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260924 19:35:14.726635 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:14.726337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:57657)
0924 19:35:14.726526 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:14.726584 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":87,"threads_started":1}
I20260924 19:35:14.805085 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:14.804829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:53041)
0924 19:35:14.804971 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:14.805022 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260924 19:35:14.926312 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:14.925965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:50693)
0924 19:35:14.926177 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:14.926244 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
I20260924 19:35:15.727454 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:15.727154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.1:33585)
0924 19:35:15.727330 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:15.727386 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260924 19:35:15.805830 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:15.805591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.2:51839)
0924 19:35:15.805738 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:15.805781 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":59,"threads_started":1}
I20260924 19:35:15.925454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 16834
I20260924 19:35:15.927057 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:15.926839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40655 (local address 127.0.98.3:47269)
0924 19:35:15.926970 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:15.927007 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260924 19:35:15.929785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 17002
I20260924 19:35:15.934353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 17135
2026-09-24T19:35:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58327 ms)
[----------] 30 tests from SecurityITest (225818 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:15 dist-test-slave-m8jb krb5kdc[17323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:15 dist-test-slave-m8jb krb5kdc[17323](info): set up 2 sockets
Sep 24 19:35:15 dist-test-slave-m8jb krb5kdc[17323](info): commencing operation
krb5kdc: starting...
W20260924 19:35:16.884001 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:16 dist-test-slave-m8jb krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:16.999500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34215
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:17.077718 17336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:17.078018 17336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:17.078097 17336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:17.079420 17336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:17.079476 17336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:17.079499 17336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:17.079517 17336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:17.079537 17336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:17.081142 17336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:17.081394 17336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:17.081606 17336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:17.086241 17336 server_base.cc:1034] running on GCE node
W20260924 19:35:17.086193 17342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:17.086206 17344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:17.086323 17341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:17.086751 17336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:17.087039 17336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:17.088198 17336 hybrid_clock.cc:648] HybridClock initialized: now 1790278517088187 us; error 29 us; skew 500 ppm
Sep 24 19:35:17 dist-test-slave-m8jb krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:17.093845 17336 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:17.094280 17336 webserver.cc:460] Webserver started at http://127.0.98.62:37117/ using document root <none> and password file <none>
I20260924 19:35:17.094509 17336 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:17.094553 17336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:17.094655 17336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:17.095707 17336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "4e2bcacec0dd43cbbb515da6d63291f7"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "da141156b867b1d747a93f326692e5f7"
server_key_iv: "f635fc41ccd7fdeab87681482d10e250"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.096014 17336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e2bcacec0dd43cbbb515da6d63291f7"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "da141156b867b1d747a93f326692e5f7"
server_key_iv: "f635fc41ccd7fdeab87681482d10e250"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.097275 17336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:35:17.097944 17351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.098105 17336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:17.098184 17336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "4e2bcacec0dd43cbbb515da6d63291f7"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "da141156b867b1d747a93f326692e5f7"
server_key_iv: "f635fc41ccd7fdeab87681482d10e250"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.098238 17336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:17.116557 17336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:17.118705 17336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:17.118813 17336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:17.122484 17336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34215
I20260924 19:35:17.122543 17403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34215 every 8 connection(s)
I20260924 19:35:17.122841 17336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:17.123647 17404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:17.124079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 17336
I20260924 19:35:17.124172 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:17.124359 392 external_mini_cluster.cc:1428] Setting key f03e3b7c924d9bfd6d8315184cb8cfdd
I20260924 19:35:17.126430 17404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7: Bootstrap starting.
I20260924 19:35:17.126962 17404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:17.127226 17404 log.cc:824] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:17.127806 17404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7: No bootstrap required, opened a new log
I20260924 19:35:17.129791 17404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2bcacec0dd43cbbb515da6d63291f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } }
I20260924 19:35:17.129935 17404 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:17.129977 17404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e2bcacec0dd43cbbb515da6d63291f7, State: Initialized, Role: FOLLOWER
I20260924 19:35:17.130102 17404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2bcacec0dd43cbbb515da6d63291f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } }
I20260924 19:35:17.130177 17404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:17.130216 17404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:17.130263 17404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 19:35:17 dist-test-slave-m8jb krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:17.130856 17404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2bcacec0dd43cbbb515da6d63291f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } }
I20260924 19:35:17.130950 17404 leader_election.cc:302] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e2bcacec0dd43cbbb515da6d63291f7; no voters:
I20260924 19:35:17.131099 17404 leader_election.cc:288] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:17.131266 17410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:17.131361 17404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:17.131400 17410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [term 1 LEADER]: Becoming Leader. State: Replica: 4e2bcacec0dd43cbbb515da6d63291f7, State: Running, Role: LEADER
I20260924 19:35:17.131561 17410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2bcacec0dd43cbbb515da6d63291f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } }
I20260924 19:35:17.132014 17411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e2bcacec0dd43cbbb515da6d63291f7. Latest consensus state: current_term: 1 leader_uuid: "4e2bcacec0dd43cbbb515da6d63291f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2bcacec0dd43cbbb515da6d63291f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } } }
I20260924 19:35:17.132170 17411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:17.132352 17409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e2bcacec0dd43cbbb515da6d63291f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e2bcacec0dd43cbbb515da6d63291f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34215 } } }
I20260924 19:35:17.132475 17409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:17.132648 17418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:17.133164 17418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:17.134274 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.124868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.0.98.62:34215)
0924 19:35:17.125038 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:17.125047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:17.125071 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:35:17.125467 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:17.125474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:17.125516 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:17.125623 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:17.126362 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.127379 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.128136 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.128538 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.128653 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.130849 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:17.130853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:17.130854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:17.130865 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:17.132847 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.133191 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.133193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.133194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.133297 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.133469 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.133470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.133470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.133704 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:17.133844 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:17.133955 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:35:17.134083 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260924 19:35:17.134953 17418 catalog_manager.cc:1347] Generated new cluster ID: 0d1fd25b281d415a8d33a80e0ddd80d0
I20260924 19:35:17.135004 17418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:17.143247 17418 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:17.143930 17418 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:17.149852 17418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7: Generated new TSK 0
I20260924 19:35:17.150030 17418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:35:17.179670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34215
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:17.259346 17430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:17.259541 17430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:17.259562 17430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:17.260682 17430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:17.260716 17430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:17.260739 17430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:35:17.262127 17430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:17.262301 17430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:17.262475 17430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:17.266893 17435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:17.266983 17436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:17.267004 17438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:17.267321 17430 server_base.cc:1034] running on GCE node
I20260924 19:35:17.267521 17430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:17.267712 17430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:17.268842 17430 hybrid_clock.cc:648] HybridClock initialized: now 1790278517268829 us; error 29 us; skew 500 ppm
Sep 24 19:35:17 dist-test-slave-m8jb krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:17.271140 17430 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:35:17.271559 17430 webserver.cc:460] Webserver started at http://127.0.98.1:40745/ using document root <none> and password file <none>
I20260924 19:35:17.271740 17430 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:17.271777 17430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:17.271864 17430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:17.272856 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "0e134f91e1dc46cba8c064b869989806"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "b255d6eb11eb79aab92b1aa24ff31cab"
server_key_iv: "7bdc00aa208c6cbce66c9b894f192eb1"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.273152 17430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0e134f91e1dc46cba8c064b869989806"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "b255d6eb11eb79aab92b1aa24ff31cab"
server_key_iv: "7bdc00aa208c6cbce66c9b894f192eb1"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.274362 17430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:17.275043 17445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.275228 17430 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:17.275314 17430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "0e134f91e1dc46cba8c064b869989806"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "b255d6eb11eb79aab92b1aa24ff31cab"
server_key_iv: "7bdc00aa208c6cbce66c9b894f192eb1"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.275384 17430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:17.287539 17430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:17.289491 17430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:17.289613 17430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:17.289793 17430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:17.290069 17430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:17.290102 17430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.290129 17430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:17.290153 17430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.295527 17430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45737
I20260924 19:35:17.295579 17558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45737 every 8 connection(s)
I20260924 19:35:17.295872 17430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:35:17 dist-test-slave-m8jb krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:17.304198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 17430
I20260924 19:35:17.304302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:35:17.304563 392 external_mini_cluster.cc:1428] Setting key 987ffcc13bc153809301308865d93681
I20260924 19:35:17.304785 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.296538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33337 (local address 127.0.98.62:34215)
0924 19:35:17.296599 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:17.296601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:17.296837 (+ 236us) server_negotiation.cc:407] Connection header received
0924 19:35:17.297914 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:17.297915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:17.297929 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:17.297979 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:17.298540 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.299041 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.299979 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.300197 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.300249 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.302195 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:17.302196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:17.302196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:17.302202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:17.303282 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.303682 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.303683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.303684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.303753 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.304064 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.304065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.304065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.304185 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:17.304235 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:17.304580 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 19:35:17.304701 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:17.304926 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.296539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34215 (local address 127.0.98.1:33337)
0924 19:35:17.296719 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:17.296763 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:17.297813 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:17.298041 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:17.298045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:17.298278 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:17.298457 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.298460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.299123 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.299124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.299902 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.299904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.300261 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.300261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.300381 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.300873 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:17.300881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:17.302114 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:17.303366 (+ 1252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:17.303373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:17.303375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:17.303590 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:17.303841 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:17.303843 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:17.303843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:17.303968 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:17.304241 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:17.304247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:17.304409 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 19:35:17.304528 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 19:35:17.304719 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260924 19:35:17.305297 17559 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34215
I20260924 19:35:17.305394 17559 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:17.305553 17559 heartbeater.cc:511] Master 127.0.98.62:34215 requested a full tablet report, sending...
I20260924 19:35:17.306075 17368 ts_manager.cc:194] Registered new tserver with Master: 0e134f91e1dc46cba8c064b869989806 (127.0.98.1:45737)
I20260924 19:35:17.306861 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33337
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:35:17.329727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34215
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:17.407286 17564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:17.407495 17564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:17.407527 17564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:17.408702 17564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:17.408739 17564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:17.408774 17564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:35:17.410218 17564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:17.410440 17564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:17.410629 17564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:17.415376 17564 server_base.cc:1034] running on GCE node
W20260924 19:35:17.415354 17570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:17.415360 17572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:17.415406 17569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:17.415756 17564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:17.415970 17564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:17.417106 17564 hybrid_clock.cc:648] HybridClock initialized: now 1790278517417089 us; error 31 us; skew 500 ppm
Sep 24 19:35:17 dist-test-slave-m8jb krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:17.419409 17564 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:35:17.419813 17564 webserver.cc:460] Webserver started at http://127.0.98.2:34847/ using document root <none> and password file <none>
I20260924 19:35:17.419997 17564 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:17.420042 17564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:17.420157 17564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:17.421191 17564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "8012997a220f4763b72a01b8841e6dd3"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "44caf6e730f3776644ae59909028a178"
server_key_iv: "392f1f2f190346f7df61994369aad1ea"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.421502 17564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8012997a220f4763b72a01b8841e6dd3"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "44caf6e730f3776644ae59909028a178"
server_key_iv: "392f1f2f190346f7df61994369aad1ea"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.422809 17564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:35:17.423460 17579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.423671 17564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:17.423745 17564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "8012997a220f4763b72a01b8841e6dd3"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "44caf6e730f3776644ae59909028a178"
server_key_iv: "392f1f2f190346f7df61994369aad1ea"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.423813 17564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:17.441028 17564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:17.442884 17564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:17.442996 17564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:17.443236 17564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:17.443526 17564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:17.443567 17564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.443598 17564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:17.443614 17564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.448765 17564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36333
I20260924 19:35:17.448948 17692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36333 every 8 connection(s)
I20260924 19:35:17.449103 17564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:35:17.454083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 17564
I20260924 19:35:17.454178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:35:17.454349 392 external_mini_cluster.cc:1428] Setting key 6ee0dccd1ad95d4c6e8473baba028b52
Sep 24 19:35:17 dist-test-slave-m8jb krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:17.459028 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.449754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38869 (local address 127.0.98.62:34215)
0924 19:35:17.449797 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:17.449799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:17.449983 (+ 184us) server_negotiation.cc:407] Connection header received
0924 19:35:17.450885 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:17.450886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:17.450921 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:17.450970 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:17.451619 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.452116 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.453090 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.453273 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.453326 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.456437 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:17.456439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:17.456439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:17.456447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:17.457839 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.458182 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.458183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.458183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.458244 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.458430 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.458431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.458431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.458565 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:17.458624 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:17.458855 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 19:35:17.458937 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:35:17.459095 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.449689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34215 (local address 127.0.98.2:38869)
0924 19:35:17.449861 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:17.449902 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:17.450776 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:17.451051 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:17.451055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:17.451304 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:17.451528 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.451532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.452192 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.452193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.453009 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.453011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.453343 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.453345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.453473 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.453887 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:17.453894 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:17.456338 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:17.457909 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:17.457915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:17.457916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:17.458117 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:17.458297 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:17.458297 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:17.458297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:17.458380 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:17.458622 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:17.458625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:17.458731 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 19:35:17.458802 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:35:17.458915 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":67,"threads_started":1}
I20260924 19:35:17.459507 17693 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34215
I20260924 19:35:17.459599 17693 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:17.459758 17693 heartbeater.cc:511] Master 127.0.98.62:34215 requested a full tablet report, sending...
I20260924 19:35:17.460121 17368 ts_manager.cc:194] Registered new tserver with Master: 8012997a220f4763b72a01b8841e6dd3 (127.0.98.2:36333)
I20260924 19:35:17.460611 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38869
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:35:17.480034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34215
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:17.558650 17698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:17.558844 17698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:17.558866 17698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:17.560024 17698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:17.560066 17698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:17.560092 17698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:35:17.561568 17698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:17.561779 17698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:17.561985 17698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:17.566728 17704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:17.566912 17698 server_base.cc:1034] running on GCE node
W20260924 19:35:17.566876 17706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:17.566737 17703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:17.567216 17698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:17.567456 17698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:17.568617 17698 hybrid_clock.cc:648] HybridClock initialized: now 1790278517568605 us; error 24 us; skew 500 ppm
Sep 24 19:35:17 dist-test-slave-m8jb krb5kdc[17323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:17.570981 17698 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:35:17.571390 17698 webserver.cc:460] Webserver started at http://127.0.98.3:46185/ using document root <none> and password file <none>
I20260924 19:35:17.571614 17698 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:17.571661 17698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:17.571782 17698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:17.572853 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a7f87537c6d43939179fc55b05b5fba"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "6e42ac56dedc3c9b498ade273055c327"
server_key_iv: "79c0f0f51f96958afef1dff07b2578e5"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.573180 17698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a7f87537c6d43939179fc55b05b5fba"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "6e42ac56dedc3c9b498ade273055c327"
server_key_iv: "79c0f0f51f96958afef1dff07b2578e5"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.574450 17698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:17.575188 17713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.575335 17698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:17.575424 17698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "4a7f87537c6d43939179fc55b05b5fba"
format_stamp: "Formatted at 2026-09-24 19:35:17 on dist-test-slave-m8jb"
server_key: "6e42ac56dedc3c9b498ade273055c327"
server_key_iv: "79c0f0f51f96958afef1dff07b2578e5"
server_key_version: "encryptionkey@0"
I20260924 19:35:17.575481 17698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:17.591104 17698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:17.593237 17698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:17.593376 17698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:17.593596 17698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:17.593883 17698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:17.593915 17698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.593945 17698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:17.593964 17698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:17.599462 17698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38755
I20260924 19:35:17.599526 17826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38755 every 8 connection(s)
I20260924 19:35:17.599810 17698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:35:17.604954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 17698
I20260924 19:35:17.605062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:35:17.605247 392 external_mini_cluster.cc:1428] Setting key 4468867cf4f616b163a0f40d1a7fe90d
Sep 24 19:35:17 dist-test-slave-m8jb krb5kdc[17323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:17.609207 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.600484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35425 (local address 127.0.98.62:34215)
0924 19:35:17.600559 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:17.600561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:17.600743 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:35:17.601744 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:17.601745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:17.601755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:17.601807 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:17.602441 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.602936 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.603894 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.604100 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.604168 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.606389 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:17.606390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:17.606390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:17.606396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:17.607768 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.608216 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.608217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.608217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.608269 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.608514 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.608514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.608515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.608661 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:17.608722 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:17.609029 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 19:35:17.609117 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:35:17.609401 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.600461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34215 (local address 127.0.98.3:35425)
0924 19:35:17.600601 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:17.600659 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:17.601644 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:17.601861 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:17.601864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:17.602117 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:17.602308 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.602311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.603006 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.603007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.603802 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.603804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.604168 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.604168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.604259 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.604847 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:17.604855 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:17.606306 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:17.607846 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:17.607854 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:17.607856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:17.608149 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:17.608326 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:17.608326 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:17.608327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:17.608451 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:17.608714 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:17.608720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:17.608883 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 19:35:17.609009 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 19:35:17.609173 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260924 19:35:17.609817 17827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34215
I20260924 19:35:17.609910 17827 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:17.610110 17827 heartbeater.cc:511] Master 127.0.98.62:34215 requested a full tablet report, sending...
I20260924 19:35:17.610512 17368 ts_manager.cc:194] Registered new tserver with Master: 4a7f87537c6d43939179fc55b05b5fba (127.0.98.3:38755)
I20260924 19:35:17.610982 17368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35425
I20260924 19:35:17.616719 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:35:17.623035 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.617964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.0.98.62:34215)
0924 19:35:17.618021 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:17.618023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:17.618100 (+ 77us) server_negotiation.cc:407] Connection header received
0924 19:35:17.618140 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:17.618140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:17.618149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:17.618190 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:17.618666 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.619151 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.619900 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.620126 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.620175 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.621173 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:17.621174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:17.621175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:17.621180 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:17.622207 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.622473 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.622474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.622474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.622515 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:17.622621 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:17.622622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:17.622622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:17.622741 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:17.622807 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:17.622881 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 19:35:17.622938 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:35:17.624287 17368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:35:17.625844 17368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:35:17.631312 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.627838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36333 (local address 127.0.0.1:51656)
0924 19:35:17.628100 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:17.628135 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:17.628203 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:17.628426 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:17.628430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:17.628432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:17.628493 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.628495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.629423 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.629424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.630231 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.630233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.631030 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.631031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.631109 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.631124 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:35:17.631186 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:35:17.631238 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":59,"threads_started":1}
I20260924 19:35:17.631317 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.627928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51656 (local address 127.0.98.2:36333)
0924 19:35:17.628074 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:17.628079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:17.628282 (+ 203us) server_negotiation.cc:407] Connection header received
0924 19:35:17.628328 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:17.628330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:17.628346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:17.628409 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:17.628546 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.629336 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.630311 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.630941 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.631046 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.631115 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:17.631198 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:35:17.631238 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260924 19:35:17.632134 17627 tablet_service.cc:1467] Processing CreateTablet for tablet 1aaa43749f6e4c7b942bdb85cbfc423d (DEFAULT_TABLE table=test-table [id=369236f70dee4a7e984f30ec4cf3f975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:17.632478 17627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aaa43749f6e4c7b942bdb85cbfc423d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:17.632500 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.627854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38755 (local address 127.0.0.1:49266)
0924 19:35:17.628231 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:17.628241 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:17.628296 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:17.628564 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:17.628565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:17.628567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:17.628612 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.628613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.629994 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.629995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.631238 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.631239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.632232 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.632233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.632292 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.632295 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:17.632380 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:35:17.632421 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":29,"thread_start_us":47,"threads_started":1}
I20260924 19:35:17.632534 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.628064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.0.98.3:38755)
0924 19:35:17.628247 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:17.628251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:17.628306 (+ 55us) server_negotiation.cc:407] Connection header received
0924 19:35:17.628393 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:17.628396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:17.628412 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:17.628480 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:17.628789 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.629869 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.631330 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.632049 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.632119 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.632175 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:17.632404 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 19:35:17.632449 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260924 19:35:17.633221 17761 tablet_service.cc:1467] Processing CreateTablet for tablet 1aaa43749f6e4c7b942bdb85cbfc423d (DEFAULT_TABLE table=test-table [id=369236f70dee4a7e984f30ec4cf3f975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:17.633503 17761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aaa43749f6e4c7b942bdb85cbfc423d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:17.635258 17841 tablet_bootstrap.cc:492] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Bootstrap starting.
I20260924 19:35:17.635878 17842 tablet_bootstrap.cc:492] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Bootstrap starting.
I20260924 19:35:17.635955 17841 tablet_bootstrap.cc:654] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:17.636042 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.627854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45737 (local address 127.0.0.1:35370)
0924 19:35:17.628362 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:17.628369 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:17.628406 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:17.628751 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:17.628752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:17.628754 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:17.628793 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.628794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.630041 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.630042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.631216 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:17.631218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.632307 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.632307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:17.632357 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.632362 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:35:17.632401 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 19:35:17.632428 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":41,"thread_start_us":47,"threads_started":1}
I20260924 19:35:17.636229 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:17.628078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.0.98.1:45737)
0924 19:35:17.628345 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:17.628348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:17.628459 (+ 111us) server_negotiation.cc:407] Connection header received
0924 19:35:17.628501 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:17.628504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:17.628517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:17.628667 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:17.628909 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.629947 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.631304 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:17.632178 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:17.632232 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:17.632281 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:17.632463 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 19:35:17.632501 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":85,"threads_started":1}
I20260924 19:35:17.636541 17841 log.cc:824] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:17.636562 17842 tablet_bootstrap.cc:654] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:17.636924 17493 tablet_service.cc:1467] Processing CreateTablet for tablet 1aaa43749f6e4c7b942bdb85cbfc423d (DEFAULT_TABLE table=test-table [id=369236f70dee4a7e984f30ec4cf3f975]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:17.637109 17842 log.cc:824] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:17.637243 17841 tablet_bootstrap.cc:492] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: No bootstrap required, opened a new log
I20260924 19:35:17.637239 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aaa43749f6e4c7b942bdb85cbfc423d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:17.637315 17841 ts_tablet_manager.cc:1397] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:17.637832 17842 tablet_bootstrap.cc:492] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: No bootstrap required, opened a new log
I20260924 19:35:17.637899 17842 ts_tablet_manager.cc:1397] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:17.638748 17841 raft_consensus.cc:348] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:17.638885 17841 raft_consensus.cc:374] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:17.638916 17841 raft_consensus.cc:729] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012997a220f4763b72a01b8841e6dd3, State: Initialized, Role: FOLLOWER
I20260924 19:35:17.639073 17841 consensus_queue.cc:260] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:17.639521 17841 ts_tablet_manager.cc:1428] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:17.639525 17693 heartbeater.cc:503] Master 127.0.98.62:34215 was elected leader, sending a full tablet report...
I20260924 19:35:17.639778 17842 raft_consensus.cc:348] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:17.639943 17842 raft_consensus.cc:374] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:17.640018 17842 raft_consensus.cc:729] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a7f87537c6d43939179fc55b05b5fba, State: Initialized, Role: FOLLOWER
I20260924 19:35:17.640133 17842 consensus_queue.cc:260] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:17.640342 17846 tablet_bootstrap.cc:492] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806: Bootstrap starting.
I20260924 19:35:17.640378 17827 heartbeater.cc:503] Master 127.0.98.62:34215 was elected leader, sending a full tablet report...
I20260924 19:35:17.640370 17842 ts_tablet_manager.cc:1428] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:17.641099 17846 tablet_bootstrap.cc:654] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:17.641566 17846 log.cc:824] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:17.642119 17846 tablet_bootstrap.cc:492] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806: No bootstrap required, opened a new log
I20260924 19:35:17.642174 17846 ts_tablet_manager.cc:1397] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:17.643568 17846 raft_consensus.cc:348] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:17.643731 17846 raft_consensus.cc:374] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:17.643772 17846 raft_consensus.cc:729] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e134f91e1dc46cba8c064b869989806, State: Initialized, Role: FOLLOWER
I20260924 19:35:17.643877 17846 consensus_queue.cc:260] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:17.644105 17846 ts_tablet_manager.cc:1428] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:17.644132 17559 heartbeater.cc:503] Master 127.0.98.62:34215 was elected leader, sending a full tablet report...
W20260924 19:35:17.699623 17694 tablet.cc:2369] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:35:17.796598 17560 tablet.cc:2369] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:35:17.850399 17828 tablet.cc:2369] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:35:18.056066 17849 raft_consensus.cc:482] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:18.056226 17849 raft_consensus.cc:504] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:18.056587 17849 leader_election.cc:288] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012997a220f4763b72a01b8841e6dd3 (127.0.98.2:36333), 4a7f87537c6d43939179fc55b05b5fba (127.0.98.3:38755)
I20260924 19:35:18.057061 17845 raft_consensus.cc:482] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:18.057171 17845 raft_consensus.cc:504] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:18.057667 17845 leader_election.cc:288] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e134f91e1dc46cba8c064b869989806 (127.0.98.1:45737), 4a7f87537c6d43939179fc55b05b5fba (127.0.98.3:38755)
I20260924 19:35:18.060395 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.056921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51061 (local address 127.0.98.2:36333)
0924 19:35:18.056965 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.056968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.057025 (+ 57us) server_negotiation.cc:407] Connection header received
0924 19:35:18.057091 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.057093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.057103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.057148 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.057323 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.058060 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.059543 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.060046 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.060084 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.060095 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.060279 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.060315 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:35:18.060443 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.056907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51923 (local address 127.0.98.3:38755)
0924 19:35:18.056981 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.056983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.057083 (+ 100us) server_negotiation.cc:407] Connection header received
0924 19:35:18.057190 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.057191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.057199 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.057226 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.057517 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.058180 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.059534 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.060082 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.060123 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.060132 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.060334 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.060378 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:35:18.060685 17850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.056803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38755 (local address 127.0.98.1:51923)
0924 19:35:18.057011 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.057021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.057109 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.057289 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.057290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.057292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.057417 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.057419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.058266 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.058268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.059451 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.059454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.060186 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.060187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.060264 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.060268 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.060543 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.060585 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":63,"threads_started":1}
I20260924 19:35:18.060861 17851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.056794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36333 (local address 127.0.98.1:51061)
0924 19:35:18.056957 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.056969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.057034 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.057196 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.057197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.057199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.057273 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.057275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.058224 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.058224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.059439 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.059442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.060145 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.060146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.060210 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.060213 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.060739 (+ 526us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.060780 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":15,"thread_start_us":73,"threads_started":1}
I20260924 19:35:18.060923 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.057628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46261 (local address 127.0.98.1:45737)
0924 19:35:18.057661 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.057662 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.057762 (+ 100us) server_negotiation.cc:407] Connection header received
0924 19:35:18.057862 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.057862 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.057871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.057914 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.058092 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.058757 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.060095 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.060585 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.060622 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.060652 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.060852 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.060880 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:35:18.061134 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaa43749f6e4c7b942bdb85cbfc423d" candidate_uuid: "0e134f91e1dc46cba8c064b869989806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7f87537c6d43939179fc55b05b5fba" is_pre_election: true
I20260924 19:35:18.061236 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.057569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45737 (local address 127.0.98.2:46261)
0924 19:35:18.057707 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.057716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.057784 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.057952 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.057953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.057954 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.058019 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.058021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.058858 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.058858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.060007 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.060009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.060702 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.060703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.060769 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.060773 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.060956 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.060993 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":43,"threads_started":1}
I20260924 19:35:18.061281 17781 raft_consensus.cc:2454] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e134f91e1dc46cba8c064b869989806 in term 0.
I20260924 19:35:18.061466 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.057933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38755 (local address 127.0.98.2:33261)
0924 19:35:18.058179 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.058188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.058224 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.058437 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.058438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.058439 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.058495 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.058496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.059408 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.059409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.060530 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.060533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.061236 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.061237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.061304 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.061308 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.061366 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.061399 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":56,"threads_started":1}
I20260924 19:35:18.061445 17647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaa43749f6e4c7b942bdb85cbfc423d" candidate_uuid: "0e134f91e1dc46cba8c064b869989806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012997a220f4763b72a01b8841e6dd3" is_pre_election: true
I20260924 19:35:18.061479 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaa43749f6e4c7b942bdb85cbfc423d" candidate_uuid: "8012997a220f4763b72a01b8841e6dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e134f91e1dc46cba8c064b869989806" is_pre_election: true
I20260924 19:35:18.061538 17647 raft_consensus.cc:2454] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e134f91e1dc46cba8c064b869989806 in term 0.
I20260924 19:35:18.061499 17447 leader_election.cc:302] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e134f91e1dc46cba8c064b869989806, 4a7f87537c6d43939179fc55b05b5fba; no voters:
I20260924 19:35:18.061580 17513 raft_consensus.cc:2454] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012997a220f4763b72a01b8841e6dd3 in term 0.
I20260924 19:35:18.061704 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.058067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33261 (local address 127.0.98.3:38755)
0924 19:35:18.058325 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.058326 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.058333 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:18.058349 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.058350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.058359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.058551 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.058578 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.059321 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.060614 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.061172 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.061236 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.061245 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.061600 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.061640 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
I20260924 19:35:18.061743 17849 raft_consensus.cc:2781] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:35:18.061795 17849 raft_consensus.cc:482] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:35:18.061818 17849 raft_consensus.cc:3037] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:18.061847 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaa43749f6e4c7b942bdb85cbfc423d" candidate_uuid: "8012997a220f4763b72a01b8841e6dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7f87537c6d43939179fc55b05b5fba" is_pre_election: true
I20260924 19:35:18.061870 17580 leader_election.cc:302] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e134f91e1dc46cba8c064b869989806, 8012997a220f4763b72a01b8841e6dd3; no voters:
I20260924 19:35:18.061903 17781 raft_consensus.cc:2454] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012997a220f4763b72a01b8841e6dd3 in term 0.
I20260924 19:35:18.061972 17845 raft_consensus.cc:2781] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:35:18.062006 17845 raft_consensus.cc:482] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:35:18.062024 17845 raft_consensus.cc:3037] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:18.062685 17849 raft_consensus.cc:504] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:18.062700 17845 raft_consensus.cc:504] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:18.062927 17849 leader_election.cc:288] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [CANDIDATE]: Term 1 election: Requested vote from peers 8012997a220f4763b72a01b8841e6dd3 (127.0.98.2:36333), 4a7f87537c6d43939179fc55b05b5fba (127.0.98.3:38755)
I20260924 19:35:18.062983 17647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaa43749f6e4c7b942bdb85cbfc423d" candidate_uuid: "0e134f91e1dc46cba8c064b869989806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012997a220f4763b72a01b8841e6dd3"
I20260924 19:35:18.062974 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaa43749f6e4c7b942bdb85cbfc423d" candidate_uuid: "8012997a220f4763b72a01b8841e6dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7f87537c6d43939179fc55b05b5fba"
I20260924 19:35:18.062987 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaa43749f6e4c7b942bdb85cbfc423d" candidate_uuid: "0e134f91e1dc46cba8c064b869989806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a7f87537c6d43939179fc55b05b5fba"
I20260924 19:35:18.063071 17647 raft_consensus.cc:2379] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0e134f91e1dc46cba8c064b869989806 in current term 1: Already voted for candidate 8012997a220f4763b72a01b8841e6dd3 in this term.
I20260924 19:35:18.063053 17780 raft_consensus.cc:3037] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:18.063210 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aaa43749f6e4c7b942bdb85cbfc423d" candidate_uuid: "8012997a220f4763b72a01b8841e6dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e134f91e1dc46cba8c064b869989806"
I20260924 19:35:18.063285 17513 raft_consensus.cc:2379] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8012997a220f4763b72a01b8841e6dd3 in current term 1: Already voted for candidate 0e134f91e1dc46cba8c064b869989806 in this term.
I20260924 19:35:18.063302 17447 leader_election.cc:302] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0e134f91e1dc46cba8c064b869989806; no voters: 4a7f87537c6d43939179fc55b05b5fba, 8012997a220f4763b72a01b8841e6dd3
I20260924 19:35:18.063397 17845 leader_election.cc:288] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [CANDIDATE]: Term 1 election: Requested vote from peers 0e134f91e1dc46cba8c064b869989806 (127.0.98.1:45737), 4a7f87537c6d43939179fc55b05b5fba (127.0.98.3:38755)
I20260924 19:35:18.063477 17849 raft_consensus.cc:2726] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 19:35:18.063812 17780 raft_consensus.cc:2454] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012997a220f4763b72a01b8841e6dd3 in term 1.
I20260924 19:35:18.064047 17581 leader_election.cc:302] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a7f87537c6d43939179fc55b05b5fba, 8012997a220f4763b72a01b8841e6dd3; no voters: 0e134f91e1dc46cba8c064b869989806
I20260924 19:35:18.064155 17845 raft_consensus.cc:2781] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:18.064322 17845 raft_consensus.cc:686] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 8012997a220f4763b72a01b8841e6dd3, State: Running, Role: LEADER
I20260924 19:35:18.064415 17845 consensus_queue.cc:237] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } }
I20260924 19:35:18.065235 17367 catalog_manager.cc:5495] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012997a220f4763b72a01b8841e6dd3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8012997a220f4763b72a01b8841e6dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012997a220f4763b72a01b8841e6dd3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 } health_report { overall_health: UNKNOWN } } }
I20260924 19:35:18.070529 17838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.068284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.0.98.2:36333)
0924 19:35:18.068335 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.068337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.068345 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:35:18.068403 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.068404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.068420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.068476 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:18.068614 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.069111 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.069885 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.070066 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.070122 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.070237 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:18.070326 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:18.070390 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.070462 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.070478 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:35:18.072070 17513 raft_consensus.cc:1261] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 1 FOLLOWER]: Refusing update from remote peer 8012997a220f4763b72a01b8841e6dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:18.072073 17780 raft_consensus.cc:1261] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 1 FOLLOWER]: Refusing update from remote peer 8012997a220f4763b72a01b8841e6dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:18.072327 17855 consensus_queue.cc:1035] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a7f87537c6d43939179fc55b05b5fba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:18.072455 17845 consensus_queue.cc:1035] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e134f91e1dc46cba8c064b869989806" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:18.075317 17859 mvcc.cc:205] Tried to move back new op lower bound from 7332980810021048320 to 7332980809992503296. Current Snapshot: MvccSnapshot[applied={T|T < 7332980810021048320}]
I20260924 19:35:18.076772 17368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:18.076870 17368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:18.077548 17368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e2bcacec0dd43cbbb515da6d63291f7: Sending DeleteTablet for 3 replicas of tablet 1aaa43749f6e4c7b942bdb85cbfc423d
I20260924 19:35:18.077875 17761 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aaa43749f6e4c7b942bdb85cbfc423d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49266
I20260924 19:35:18.077874 17493 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aaa43749f6e4c7b942bdb85cbfc423d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35370
I20260924 19:35:18.078066 17869 tablet_replica.cc:330] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806: stopping tablet replica
I20260924 19:35:18.078068 17627 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aaa43749f6e4c7b942bdb85cbfc423d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51656
I20260924 19:35:18.078164 17869 raft_consensus.cc:2229] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:18.078312 17870 tablet_replica.cc:330] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: stopping tablet replica
I20260924 19:35:18.078405 17869 raft_consensus.cc:2258] T 1aaa43749f6e4c7b942bdb85cbfc423d P 0e134f91e1dc46cba8c064b869989806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:18.078411 17870 raft_consensus.cc:2229] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:35:18.078490 17870 raft_consensus.cc:2258] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:18.078557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 17430
I20260924 19:35:18.078902 17870 ts_tablet_manager.cc:1905] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:18.080147 17868 tablet_replica.cc:330] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: stopping tablet replica
I20260924 19:35:18.080226 17868 raft_consensus.cc:2229] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:18.080317 17868 raft_consensus.cc:2258] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:18.080665 17868 ts_tablet_manager.cc:1905] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:18.080752 17870 ts_tablet_manager.cc:1918] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:35:18.080832 17870 log.cc:1196] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/1aaa43749f6e4c7b942bdb85cbfc423d
I20260924 19:35:18.081065 17870 ts_tablet_manager.cc:1939] T 1aaa43749f6e4c7b942bdb85cbfc423d P 8012997a220f4763b72a01b8841e6dd3: Deleting consensus metadata
I20260924 19:35:18.081449 17354 catalog_manager.cc:4841] TS 8012997a220f4763b72a01b8841e6dd3 (127.0.98.2:36333): tablet 1aaa43749f6e4c7b942bdb85cbfc423d (table test-table [id=369236f70dee4a7e984f30ec4cf3f975]) successfully deleted
I20260924 19:35:18.081983 17868 ts_tablet_manager.cc:1918] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:35:18.082041 17868 log.cc:1196] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/1aaa43749f6e4c7b942bdb85cbfc423d
I20260924 19:35:18.082250 17868 ts_tablet_manager.cc:1939] T 1aaa43749f6e4c7b942bdb85cbfc423d P 4a7f87537c6d43939179fc55b05b5fba: Deleting consensus metadata
I20260924 19:35:18.082729 17353 catalog_manager.cc:4841] TS 4a7f87537c6d43939179fc55b05b5fba (127.0.98.3:38755): tablet 1aaa43749f6e4c7b942bdb85cbfc423d (table test-table [id=369236f70dee4a7e984f30ec4cf3f975]) successfully deleted
W20260924 19:35:18.084358 17352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45737 (error 108)
I20260924 19:35:18.084527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 17564
I20260924 19:35:18.085186 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.084658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45737 (local address 127.0.0.1:35372)
0924 19:35:18.085127 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.085161 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452}
W20260924 19:35:18.085300 17352 catalog_manager.cc:4584] TS 0e134f91e1dc46cba8c064b869989806 (127.0.98.1:45737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1aaa43749f6e4c7b942bdb85cbfc423d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45737: connect: Connection refused (error 111)
I20260924 19:35:18.089243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 17698
I20260924 19:35:18.093236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 17336
2026-09-24T19:35:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2159 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T19:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:18Z Disabled control of system clock
I20260924 19:35:18.118602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37061
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:18.198141 17875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:18.198372 17875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:18.198395 17875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:18.199625 17875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:18.199674 17875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:18.199688 17875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:18.199702 17875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:18.199714 17875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:18.201229 17875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:18.201514 17875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:18.201819 17875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:18.206652 17875 server_base.cc:1034] running on GCE node
W20260924 19:35:18.206594 17880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:18.206634 17883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:18.206635 17881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:18.207063 17875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:18.207325 17875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:18.208473 17875 hybrid_clock.cc:648] HybridClock initialized: now 1790278518208461 us; error 28 us; skew 500 ppm
I20260924 19:35:18.209606 17875 webserver.cc:460] Webserver started at http://127.0.98.62:39323/ using document root <none> and password file <none>
I20260924 19:35:18.209795 17875 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:18.209837 17875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:18.209929 17875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:18.210937 17875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "b72db693ef5a4b42ab5f099825503acb"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "5b32680e6cd7ce7f7d8f3c8f756e6e1b"
server_key_iv: "0cef2c73521d3eb75d54553c09304359"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.211210 17875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "b72db693ef5a4b42ab5f099825503acb"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "5b32680e6cd7ce7f7d8f3c8f756e6e1b"
server_key_iv: "0cef2c73521d3eb75d54553c09304359"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.212387 17875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:35:18.213029 17889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.213171 17875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.213248 17875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "b72db693ef5a4b42ab5f099825503acb"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "5b32680e6cd7ce7f7d8f3c8f756e6e1b"
server_key_iv: "0cef2c73521d3eb75d54553c09304359"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.213300 17875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:18.225771 17875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:18.226019 17875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:18.226112 17875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:18.229652 17875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37061
I20260924 19:35:18.229696 17941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37061 every 8 connection(s)
I20260924 19:35:18.230031 17875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:18.230924 17942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:18.232610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 17875
I20260924 19:35:18.232767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:18.232977 392 external_mini_cluster.cc:1428] Setting key 7118422446fde45557a516a55f444431
I20260924 19:35:18.233405 17942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb: Bootstrap starting.
I20260924 19:35:18.234131 17942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:18.234444 17942 log.cc:824] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:18.235178 17942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb: No bootstrap required, opened a new log
I20260924 19:35:18.237303 17942 raft_consensus.cc:348] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72db693ef5a4b42ab5f099825503acb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } }
I20260924 19:35:18.237468 17942 raft_consensus.cc:374] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:18.237500 17942 raft_consensus.cc:729] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72db693ef5a4b42ab5f099825503acb, State: Initialized, Role: FOLLOWER
I20260924 19:35:18.237646 17942 consensus_queue.cc:260] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72db693ef5a4b42ab5f099825503acb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } }
I20260924 19:35:18.237728 17942 raft_consensus.cc:388] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:18.237758 17942 raft_consensus.cc:482] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:18.237811 17942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:18.238330 17942 raft_consensus.cc:504] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72db693ef5a4b42ab5f099825503acb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } }
I20260924 19:35:18.238436 17942 leader_election.cc:302] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b72db693ef5a4b42ab5f099825503acb; no voters:
I20260924 19:35:18.238601 17942 leader_election.cc:288] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:18.238648 17947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:18.238842 17947 raft_consensus.cc:686] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [term 1 LEADER]: Becoming Leader. State: Replica: b72db693ef5a4b42ab5f099825503acb, State: Running, Role: LEADER
I20260924 19:35:18.238947 17942 sys_catalog.cc:564] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:18.238972 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.233515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.0.98.62:37061)
0924 19:35:18.233725 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.233735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.233770 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:35:18.234324 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.234332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.234412 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.234563 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:18.234765 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.236060 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.236932 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.237531 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.237645 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.238257 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:18.238261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:18.238262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:18.238471 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:18.238511 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:18.238526 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:18.238592 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.238665 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.238747 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260924 19:35:18.238986 17947 consensus_queue.cc:237] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72db693ef5a4b42ab5f099825503acb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } }
I20260924 19:35:18.239342 17947 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b72db693ef5a4b42ab5f099825503acb. Latest consensus state: current_term: 1 leader_uuid: "b72db693ef5a4b42ab5f099825503acb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72db693ef5a4b42ab5f099825503acb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } } }
I20260924 19:35:18.239349 17949 sys_catalog.cc:455] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b72db693ef5a4b42ab5f099825503acb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b72db693ef5a4b42ab5f099825503acb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37061 } } }
I20260924 19:35:18.239457 17947 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:18.239499 17949 sys_catalog.cc:458] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:18.240208 17956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:18.240936 17956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:18.242427 17956 catalog_manager.cc:1347] Generated new cluster ID: 0ec1615735674e7fb6caf87eb43ae9b8
I20260924 19:35:18.242476 17956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:18.251360 17956 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:18.251927 17956 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:18.256218 17956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb: Generated new TSK 0
I20260924 19:35:18.256438 17956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:18.263880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--builtin_ntp_servers=127.0.98.20:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:18.343581 17966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:18.343773 17966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:18.343791 17966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:18.344939 17966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:18.344977 17966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:18.345000 17966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:35:18.346318 17966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:18.346547 17966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:18.346815 17966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:18.351583 17972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:18.351683 17974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:18.351562 17971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:18.351818 17966 server_base.cc:1034] running on GCE node
I20260924 19:35:18.352115 17966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:18.352435 17966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:18.353616 17966 hybrid_clock.cc:648] HybridClock initialized: now 1790278518353593 us; error 37 us; skew 500 ppm
I20260924 19:35:18.354954 17966 webserver.cc:460] Webserver started at http://127.0.98.1:40093/ using document root <none> and password file <none>
I20260924 19:35:18.355211 17966 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:18.355265 17966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:18.355396 17966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:18.356792 17966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "e445ff2d664c4c98a898ade6bfa7a0cf"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "ac8d86baa19ace5753e22b476f8ed953"
server_key_iv: "e53684a5f7816b78a7780c957e094a55"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.357163 17966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e445ff2d664c4c98a898ade6bfa7a0cf"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "ac8d86baa19ace5753e22b476f8ed953"
server_key_iv: "e53684a5f7816b78a7780c957e094a55"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.358723 17966 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:35:18.359606 17980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.359772 17966 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.359862 17966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "e445ff2d664c4c98a898ade6bfa7a0cf"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "ac8d86baa19ace5753e22b476f8ed953"
server_key_iv: "e53684a5f7816b78a7780c957e094a55"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.359931 17966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:18.375267 17966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:18.375542 17966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:18.375640 17966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:18.375823 17966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:18.376078 17966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:18.376106 17966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.376130 17966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:18.376147 17966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.381573 17966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36661
I20260924 19:35:18.381645 18093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36661 every 8 connection(s)
I20260924 19:35:18.381922 17966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:35:18.387033 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.382662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60647 (local address 127.0.98.62:37061)
0924 19:35:18.382716 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.382718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.382943 (+ 225us) server_negotiation.cc:407] Connection header received
0924 19:35:18.383481 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.383482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.383494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.383545 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:18.384238 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.384805 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.385737 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.385945 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.385999 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.386776 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:18.386778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:18.386778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:18.386783 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:18.386798 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:18.386802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:18.386845 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.386926 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.386951 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:35:18.387240 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.382627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37061 (local address 127.0.98.1:60647)
0924 19:35:18.382815 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.382847 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.383350 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.383625 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.383628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.383636 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:18.384148 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.384152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.384884 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.384886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.385659 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.385661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.386018 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.386019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.386139 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.386644 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:18.386645 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:18.386673 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:18.386697 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:18.386852 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:18.386856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:18.386860 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.386924 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.387026 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260924 19:35:18.387750 18094 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37061
I20260924 19:35:18.387831 18094 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:18.388005 18094 heartbeater.cc:511] Master 127.0.98.62:37061 requested a full tablet report, sending...
I20260924 19:35:18.388501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 17966
I20260924 19:35:18.388507 17906 ts_manager.cc:194] Registered new tserver with Master: e445ff2d664c4c98a898ade6bfa7a0cf (127.0.98.1:36661)
I20260924 19:35:18.388581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:35:18.388785 392 external_mini_cluster.cc:1428] Setting key 86a7ac908bb0e47d79c8016d45a4f379
I20260924 19:35:18.389286 17906 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60647
I20260924 19:35:18.389706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--builtin_ntp_servers=127.0.98.20:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:18.467455 18097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:18.467661 18097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:18.467682 18097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:18.468817 18097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:18.468855 18097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:18.468879 18097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:35:18.470181 18097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:18.470386 18097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:18.470630 18097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:18.475412 18105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:18.475404 18102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:18.475407 18103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:18.475664 18097 server_base.cc:1034] running on GCE node
I20260924 19:35:18.475890 18097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:18.476126 18097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:18.477262 18097 hybrid_clock.cc:648] HybridClock initialized: now 1790278518477251 us; error 28 us; skew 500 ppm
I20260924 19:35:18.478432 18097 webserver.cc:460] Webserver started at http://127.0.98.2:41503/ using document root <none> and password file <none>
I20260924 19:35:18.478636 18097 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:18.478683 18097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:18.478816 18097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:18.479842 18097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "214d941d1e154922b66def49c6c9c9dd"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "d83f2746f550441baa5f854ab66a889c"
server_key_iv: "5c4ba7e6b43d336d5f477ab3c0b1c1ba"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.480155 18097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "214d941d1e154922b66def49c6c9c9dd"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "d83f2746f550441baa5f854ab66a889c"
server_key_iv: "5c4ba7e6b43d336d5f477ab3c0b1c1ba"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.481410 18097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:18.482105 18111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.482290 18097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:18.482389 18097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "214d941d1e154922b66def49c6c9c9dd"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "d83f2746f550441baa5f854ab66a889c"
server_key_iv: "5c4ba7e6b43d336d5f477ab3c0b1c1ba"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.482455 18097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:18.521178 18097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:18.521478 18097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:18.521608 18097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:18.521829 18097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:18.522136 18097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:18.522169 18097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.522197 18097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:18.522221 18097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.527438 18097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35761
I20260924 19:35:18.527480 18224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35761 every 8 connection(s)
I20260924 19:35:18.527781 18097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:35:18.532953 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.528515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52403 (local address 127.0.98.62:37061)
0924 19:35:18.528581 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.528583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.528807 (+ 224us) server_negotiation.cc:407] Connection header received
0924 19:35:18.529313 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.529314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.529325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.529388 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:18.530056 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.530600 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.531599 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.531785 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.531842 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.532684 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:18.532685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:18.532685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:18.532691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:18.532708 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:18.532712 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:18.532769 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.532868 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.532891 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:35:18.533095 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.528474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37061 (local address 127.0.98.2:52403)
0924 19:35:18.528684 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.528725 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.529199 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.529454 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.529457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.529466 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:18.529960 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.529964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.530688 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.530689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.531515 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.531517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.531875 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.531876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.531999 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.532534 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:18.532534 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:18.532562 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:18.532585 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:18.532770 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:18.532773 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:18.532775 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.532822 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.532902 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260924 19:35:18.533579 18225 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37061
I20260924 19:35:18.533671 18225 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:18.533839 18225 heartbeater.cc:511] Master 127.0.98.62:37061 requested a full tablet report, sending...
I20260924 19:35:18.534201 17906 ts_manager.cc:194] Registered new tserver with Master: 214d941d1e154922b66def49c6c9c9dd (127.0.98.2:35761)
I20260924 19:35:18.534313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 18097
I20260924 19:35:18.534406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:35:18.534579 392 external_mini_cluster.cc:1428] Setting key f2150d6cdf7a6e318075af609c40a2b6
I20260924 19:35:18.534650 17906 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52403
I20260924 19:35:18.535447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--builtin_ntp_servers=127.0.98.20:38551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:18.613088 18228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:18.613283 18228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:18.613304 18228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:18.614464 18228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:18.614504 18228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:18.614528 18228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:35:18.615942 18228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:18.616159 18228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:18.616456 18228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:18.621371 18233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:18.621377 18236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:18.621409 18234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:18.621515 18228 server_base.cc:1034] running on GCE node
I20260924 19:35:18.621917 18228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:18.622134 18228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:18.623275 18228 hybrid_clock.cc:648] HybridClock initialized: now 1790278518623243 us; error 46 us; skew 500 ppm
I20260924 19:35:18.624446 18228 webserver.cc:460] Webserver started at http://127.0.98.3:45665/ using document root <none> and password file <none>
I20260924 19:35:18.624686 18228 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:18.624740 18228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:18.624866 18228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:18.625855 18228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "16df9bd18da14651a32644b41866bd5d"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "ff727276d74553eb28896a901d609e58"
server_key_iv: "b35aefb303cc75628ce5bfda08b3e0e7"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.626173 18228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "16df9bd18da14651a32644b41866bd5d"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "ff727276d74553eb28896a901d609e58"
server_key_iv: "b35aefb303cc75628ce5bfda08b3e0e7"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.627302 18228 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 19:35:18.627949 18242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.628099 18228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:18.628173 18228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "16df9bd18da14651a32644b41866bd5d"
format_stamp: "Formatted at 2026-09-24 19:35:18 on dist-test-slave-m8jb"
server_key: "ff727276d74553eb28896a901d609e58"
server_key_iv: "b35aefb303cc75628ce5bfda08b3e0e7"
server_key_version: "encryptionkey@0"
I20260924 19:35:18.628239 18228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:18.636288 18228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:18.636567 18228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:18.636731 18228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:18.636950 18228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:18.637260 18228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:18.637301 18228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.637333 18228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:18.637359 18228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:18.643510 18228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35729
I20260924 19:35:18.643545 18355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35729 every 8 connection(s)
I20260924 19:35:18.643879 18228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:35:18.648990 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.644596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44513 (local address 127.0.98.62:37061)
0924 19:35:18.644668 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.644670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.644924 (+ 254us) server_negotiation.cc:407] Connection header received
0924 19:35:18.645413 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.645414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.645425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.645482 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:18.646154 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.646651 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.647673 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.647879 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.647964 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.648760 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:18.648762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:18.648762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:18.648766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:18.648795 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:18.648799 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:18.648840 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.648907 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.648929 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:35:18.649178 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.644582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37061 (local address 127.0.98.3:44513)
0924 19:35:18.644788 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.644823 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.645309 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.645539 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.645543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.645551 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:18.646052 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.646056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.646758 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.646759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.647574 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.647576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.647975 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.647976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.648094 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.648589 (+ 495us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:18.648589 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:18.648617 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:18.648677 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:18.648849 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:18.648853 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:18.648855 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.648906 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.648986 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260924 19:35:18.649610 18356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37061
I20260924 19:35:18.649711 18356 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:18.649803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 18228
I20260924 19:35:18.649904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:35:18.649936 18356 heartbeater.cc:511] Master 127.0.98.62:37061 requested a full tablet report, sending...
I20260924 19:35:18.650115 392 external_mini_cluster.cc:1428] Setting key d558585cfd6f79c102a340ba374ab472
I20260924 19:35:18.650395 17906 ts_manager.cc:194] Registered new tserver with Master: 16df9bd18da14651a32644b41866bd5d (127.0.98.3:35729)
I20260924 19:35:18.650861 17906 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44513
I20260924 19:35:18.651070 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:35:18.655305 17945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.652562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.0.98.62:37061)
0924 19:35:18.652667 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.652669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.652680 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:35:18.652744 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.652744 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.652753 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.652809 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:18.652968 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.653457 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.654215 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.654447 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.654501 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.655079 (+ 578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:18.655080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:18.655080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:18.655084 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:18.655099 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:18.655102 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:18.655145 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.655189 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.655209 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 19:35:18.656891 17906 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:35:18.658416 17906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:35:18.664724 18365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.660544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35761 (local address 127.0.0.1:58424)
0924 19:35:18.660745 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.660785 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.660847 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.661077 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.661080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.661081 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.661154 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.661156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.662414 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.662415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.663664 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.663667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.664418 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.664419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.664491 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.664503 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.664567 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.664618 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":35,"thread_start_us":133,"threads_started":1}
I20260924 19:35:18.664798 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.660809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.0.98.1:36661)
0924 19:35:18.661173 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.661176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.661186 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:35:18.661222 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.661224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.661238 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.661313 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.661473 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.662239 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.663750 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.664454 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.664539 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.664649 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.664676 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.664712 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":83,"threads_started":1}
I20260924 19:35:18.664978 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.661010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.0.98.3:35729)
0924 19:35:18.661376 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.661380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.661389 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:18.661421 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.661423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.661435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.661501 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.661681 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.662653 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.663694 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.664444 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.664512 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.664597 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.664862 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.664906 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":60,"threads_started":1}
I20260924 19:35:18.665249 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.660746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.0.98.2:35761)
0924 19:35:18.660939 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.660943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.660952 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:18.660989 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.660990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.661002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.661063 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.661235 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.662339 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.663788 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.664347 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.664411 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.664504 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.665158 (+ 654us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.665190 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260924 19:35:18.664801 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.660546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36661 (local address 127.0.0.1:35308)
0924 19:35:18.660749 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.660785 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.660847 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.661367 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.661367 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.661369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.661407 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.661409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.662310 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.662311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.663664 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.663667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.664546 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.664547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.664602 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.664605 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.664665 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.664699 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":29,"thread_start_us":69,"threads_started":1}
I20260924 19:35:18.664893 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.660546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35729 (local address 127.0.0.1:60206)
0924 19:35:18.661147 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.661156 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.661194 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.661564 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.661565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.661567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.661606 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.661608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.662726 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.662727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.663624 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.663626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.664648 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.664649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.664710 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.664714 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.664766 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.664805 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":2,"thread_start_us":120,"threads_started":1}
I20260924 19:35:18.666019 18028 tablet_service.cc:1467] Processing CreateTablet for tablet 48a8529893704cceaac2fb379607690b (DEFAULT_TABLE table=test-table [id=c8022c28a4ca499aa11ed55b1ef382ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:18.666019 18159 tablet_service.cc:1467] Processing CreateTablet for tablet 48a8529893704cceaac2fb379607690b (DEFAULT_TABLE table=test-table [id=c8022c28a4ca499aa11ed55b1ef382ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:18.666368 18028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a8529893704cceaac2fb379607690b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:18.666375 18159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a8529893704cceaac2fb379607690b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:18.666404 18290 tablet_service.cc:1467] Processing CreateTablet for tablet 48a8529893704cceaac2fb379607690b (DEFAULT_TABLE table=test-table [id=c8022c28a4ca499aa11ed55b1ef382ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:18.666711 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a8529893704cceaac2fb379607690b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:18.669158 18370 tablet_bootstrap.cc:492] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: Bootstrap starting.
I20260924 19:35:18.669502 18372 tablet_bootstrap.cc:492] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf: Bootstrap starting.
I20260924 19:35:18.669893 18370 tablet_bootstrap.cc:654] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:18.670136 18372 tablet_bootstrap.cc:654] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:18.670190 18370 log.cc:824] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:18.670428 18372 log.cc:824] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:18.670717 18371 tablet_bootstrap.cc:492] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: Bootstrap starting.
I20260924 19:35:18.670893 18370 tablet_bootstrap.cc:492] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: No bootstrap required, opened a new log
I20260924 19:35:18.670959 18370 ts_tablet_manager.cc:1397] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:18.671098 18372 tablet_bootstrap.cc:492] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf: No bootstrap required, opened a new log
I20260924 19:35:18.671167 18372 ts_tablet_manager.cc:1397] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:18.671324 18371 tablet_bootstrap.cc:654] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:18.671617 18371 log.cc:824] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:18.672472 18371 tablet_bootstrap.cc:492] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: No bootstrap required, opened a new log
I20260924 19:35:18.672578 18371 ts_tablet_manager.cc:1397] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:18.673306 18372 raft_consensus.cc:348] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.673449 18372 raft_consensus.cc:374] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:18.673480 18372 raft_consensus.cc:729] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e445ff2d664c4c98a898ade6bfa7a0cf, State: Initialized, Role: FOLLOWER
I20260924 19:35:18.673460 18370 raft_consensus.cc:348] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.673614 18370 raft_consensus.cc:374] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:18.673625 18372 consensus_queue.cc:260] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.673683 18370 raft_consensus.cc:729] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16df9bd18da14651a32644b41866bd5d, State: Initialized, Role: FOLLOWER
I20260924 19:35:18.673815 18370 consensus_queue.cc:260] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.673882 18372 ts_tablet_manager.cc:1428] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:35:18.674006 18094 heartbeater.cc:503] Master 127.0.98.62:37061 was elected leader, sending a full tablet report...
I20260924 19:35:18.674057 18370 ts_tablet_manager.cc:1428] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:35:18.674154 18356 heartbeater.cc:503] Master 127.0.98.62:37061 was elected leader, sending a full tablet report...
I20260924 19:35:18.674997 18371 raft_consensus.cc:348] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.675149 18371 raft_consensus.cc:374] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:18.675192 18371 raft_consensus.cc:729] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214d941d1e154922b66def49c6c9c9dd, State: Initialized, Role: FOLLOWER
I20260924 19:35:18.675285 18371 consensus_queue.cc:260] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.675462 18371 ts_tablet_manager.cc:1428] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:35:18.675535 18225 heartbeater.cc:503] Master 127.0.98.62:37061 was elected leader, sending a full tablet report...
I20260924 19:35:18.771770 18377 raft_consensus.cc:482] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:18.771900 18377 raft_consensus.cc:504] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.772212 18377 leader_election.cc:288] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214d941d1e154922b66def49c6c9c9dd (127.0.98.2:35761), e445ff2d664c4c98a898ade6bfa7a0cf (127.0.98.1:36661)
I20260924 19:35:18.775019 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.772509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45147 (local address 127.0.98.2:35761)
0924 19:35:18.772551 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.772553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.772562 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:18.772593 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.772593 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.772603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.772678 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.772881 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.773418 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.774366 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.774707 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.774757 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.774767 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.774909 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.774949 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:35:18.775028 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.772472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52035 (local address 127.0.98.1:36661)
0924 19:35:18.772520 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.772522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.772681 (+ 159us) server_negotiation.cc:407] Connection header received
0924 19:35:18.772735 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.772736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.772749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.772807 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:18.773019 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.773471 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.774378 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.774723 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.774782 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.774791 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.774923 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.774956 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:35:18.775048 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.772404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36661 (local address 127.0.98.3:52035)
0924 19:35:18.772582 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.772591 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.772679 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.772874 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.772875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.772877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.772947 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.772948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.773547 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.773548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.774297 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.774299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.774792 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.774793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.774863 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.774865 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.774912 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.774948 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260924 19:35:18.775027 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.772387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35761 (local address 127.0.98.3:45147)
0924 19:35:18.772446 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.772459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:18.772507 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:18.772697 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:18.772698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:18.772700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:18.772778 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.772780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.773494 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.773495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.774280 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:18.774282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.774773 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.774775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:18.774847 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.774850 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:18.774890 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 19:35:18.774934 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":17}
I20260924 19:35:18.775419 18048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a8529893704cceaac2fb379607690b" candidate_uuid: "16df9bd18da14651a32644b41866bd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" is_pre_election: true
I20260924 19:35:18.775472 18179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a8529893704cceaac2fb379607690b" candidate_uuid: "16df9bd18da14651a32644b41866bd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214d941d1e154922b66def49c6c9c9dd" is_pre_election: true
I20260924 19:35:18.775566 18048 raft_consensus.cc:2454] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16df9bd18da14651a32644b41866bd5d in term 0.
I20260924 19:35:18.775616 18179 raft_consensus.cc:2454] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16df9bd18da14651a32644b41866bd5d in term 0.
I20260924 19:35:18.775816 18244 leader_election.cc:302] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16df9bd18da14651a32644b41866bd5d, 214d941d1e154922b66def49c6c9c9dd; no voters:
I20260924 19:35:18.775967 18377 raft_consensus.cc:2781] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:35:18.776019 18377 raft_consensus.cc:482] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:35:18.776036 18377 raft_consensus.cc:3037] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:18.776739 18377 raft_consensus.cc:504] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.776857 18377 leader_election.cc:288] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [CANDIDATE]: Term 1 election: Requested vote from peers 214d941d1e154922b66def49c6c9c9dd (127.0.98.2:35761), e445ff2d664c4c98a898ade6bfa7a0cf (127.0.98.1:36661)
I20260924 19:35:18.777007 18179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a8529893704cceaac2fb379607690b" candidate_uuid: "16df9bd18da14651a32644b41866bd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214d941d1e154922b66def49c6c9c9dd"
I20260924 19:35:18.777006 18048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a8529893704cceaac2fb379607690b" candidate_uuid: "16df9bd18da14651a32644b41866bd5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf"
I20260924 19:35:18.777102 18179 raft_consensus.cc:3037] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:18.777102 18048 raft_consensus.cc:3037] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:18.777964 18179 raft_consensus.cc:2454] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16df9bd18da14651a32644b41866bd5d in term 1.
I20260924 19:35:18.777964 18048 raft_consensus.cc:2454] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16df9bd18da14651a32644b41866bd5d in term 1.
I20260924 19:35:18.778143 18243 leader_election.cc:302] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16df9bd18da14651a32644b41866bd5d, e445ff2d664c4c98a898ade6bfa7a0cf; no voters:
I20260924 19:35:18.778235 18377 raft_consensus.cc:2781] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 1 FOLLOWER]: Leader election won for term 1
W20260924 19:35:18.778288 18226 tablet.cc:2369] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:35:18.778381 18377 raft_consensus.cc:686] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 1 LEADER]: Becoming Leader. State: Replica: 16df9bd18da14651a32644b41866bd5d, State: Running, Role: LEADER
I20260924 19:35:18.778463 18377 consensus_queue.cc:237] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } }
I20260924 19:35:18.779261 17906 catalog_manager.cc:5495] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 16df9bd18da14651a32644b41866bd5d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "16df9bd18da14651a32644b41866bd5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16df9bd18da14651a32644b41866bd5d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35729 } health_report { overall_health: HEALTHY } } }
I20260924 19:35:18.811983 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.809947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.0.98.3:35729)
0924 19:35:18.810001 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:18.810003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:18.810010 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:18.810046 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:18.810047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:18.810064 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:18.810119 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:18.810221 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.810681 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.811393 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:18.811576 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:18.811619 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:18.811730 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:18.811821 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:18.811877 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:18.811889 (+ 12us) server_negotiation.cc:299] Negotiation successful
0924 19:35:18.811905 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:35:18.813783 18179 raft_consensus.cc:1261] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 1 FOLLOWER]: Refusing update from remote peer 16df9bd18da14651a32644b41866bd5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:18.813787 18048 raft_consensus.cc:1261] T 48a8529893704cceaac2fb379607690b P e445ff2d664c4c98a898ade6bfa7a0cf [term 1 FOLLOWER]: Refusing update from remote peer 16df9bd18da14651a32644b41866bd5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:18.814105 18377 consensus_queue.cc:1035] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "214d941d1e154922b66def49c6c9c9dd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:18.814255 18380 consensus_queue.cc:1035] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e445ff2d664c4c98a898ade6bfa7a0cf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:18.816309 18383 mvcc.cc:205] Tried to move back new op lower bound from 7332980813058654208 to 7332980812917231616. Current Snapshot: MvccSnapshot[applied={T|T < 7332980813058654208}]
I20260924 19:35:18.817166 18384 mvcc.cc:205] Tried to move back new op lower bound from 7332980813058654208 to 7332980812917231616. Current Snapshot: MvccSnapshot[applied={T|T < 7332980813058654208 or (T in {7332980813058654208})}]
I20260924 19:35:18.818655 18386 mvcc.cc:205] Tried to move back new op lower bound from 7332980813058654208 to 7332980812917231616. Current Snapshot: MvccSnapshot[applied={T|T < 7332980813058654208}]
I20260924 19:35:18.818801 17905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:18.818887 17905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:18.819625 17905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b72db693ef5a4b42ab5f099825503acb: Sending DeleteTablet for 3 replicas of tablet 48a8529893704cceaac2fb379607690b
I20260924 19:35:18.819885 18290 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a8529893704cceaac2fb379607690b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:18 UTC) from {username='slave'} at 127.0.0.1:60206
I20260924 19:35:18.820044 18159 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a8529893704cceaac2fb379607690b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:18 UTC) from {username='slave'} at 127.0.0.1:58424
I20260924 19:35:18.820154 18028 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a8529893704cceaac2fb379607690b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:18 UTC) from {username='slave'} at 127.0.0.1:35308
I20260924 19:35:18.820319 18027 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a8529893704cceaac2fb379607690b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:18 UTC) from {username='slave'} at 127.0.0.1:35308
I20260924 19:35:18.820346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 17966
I20260924 19:35:18.824788 18394 tablet_replica.cc:330] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: stopping tablet replica
I20260924 19:35:18.824898 18394 raft_consensus.cc:2229] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:18.824997 18394 raft_consensus.cc:2258] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:18.825220 18393 tablet_replica.cc:330] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: stopping tablet replica
I20260924 19:35:18.825310 18393 raft_consensus.cc:2229] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:35:18.825402 18393 raft_consensus.cc:2258] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:35:18.825490 17890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36661 (error 108)
I20260924 19:35:18.825650 18394 ts_tablet_manager.cc:1905] T 48a8529893704cceaac2fb379607690b P 214d941d1e154922b66def49c6c9c9dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:18.825820 18393 ts_tablet_manager.cc:1905] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:18.825953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 18097
I20260924 19:35:18.828150 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.828018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36661 (local address 127.0.0.1:35312)
0924 19:35:18.828066 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.828116 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:35:18.828279 17890 catalog_manager.cc:4584] TS e445ff2d664c4c98a898ade6bfa7a0cf (127.0.98.1:36661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48a8529893704cceaac2fb379607690b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36661: connect: Connection refused (error 111)
I20260924 19:35:18.828434 18393 ts_tablet_manager.cc:1918] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:35:18.828495 18393 log.cc:1196] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/48a8529893704cceaac2fb379607690b
I20260924 19:35:18.828751 18393 ts_tablet_manager.cc:1939] T 48a8529893704cceaac2fb379607690b P 16df9bd18da14651a32644b41866bd5d: Deleting consensus metadata
I20260924 19:35:18.829097 17893 catalog_manager.cc:4841] TS 16df9bd18da14651a32644b41866bd5d (127.0.98.3:35729): tablet 48a8529893704cceaac2fb379607690b (table test-table [id=c8022c28a4ca499aa11ed55b1ef382ca]) successfully deleted
I20260924 19:35:18.831585 18364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:18.831453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35761 (local address 127.0.0.1:58436)
0924 19:35:18.831519 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:18.831548 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 19:35:18.831635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 18228
I20260924 19:35:18.835541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 17875
2026-09-24T19:35:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (741 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:18 dist-test-slave-m8jb krb5kdc[18399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:18 dist-test-slave-m8jb krb5kdc[18399](info): set up 2 sockets
Sep 24 19:35:18 dist-test-slave-m8jb krb5kdc[18399](info): commencing operation
krb5kdc: starting...
W20260924 19:35:19.804219 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:19 dist-test-slave-m8jb krb5kdc[18399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:19.917546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39771
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:19.998749 18412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:19.999015 18412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:19.999053 18412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:20.000263 18412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:20.000313 18412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:20.000335 18412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:20.000355 18412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:20.000373 18412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:20.001924 18412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:20.002200 18412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:20.002416 18412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:20.007383 18417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.007383 18418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.007387 18420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.007591 18412 server_base.cc:1034] running on GCE node
I20260924 19:35:20.007922 18412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:20.008206 18412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:20.009344 18412 hybrid_clock.cc:648] HybridClock initialized: now 1790278520009333 us; error 31 us; skew 500 ppm
Sep 24 19:35:20 dist-test-slave-m8jb krb5kdc[18399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:20.011685 18412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:20.012149 18412 webserver.cc:460] Webserver started at http://127.0.98.62:42157/ using document root <none> and password file <none>
I20260924 19:35:20.012344 18412 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:20.012390 18412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:20.012512 18412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:20.013602 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "cbbdd4050b2b4ff1bd3428301064fa60"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "7a97f7564f1c6bcc4aa6a13b88ad0ffa"
server_key_iv: "7229cb6047ca56aa202d2de242d54d8f"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.013949 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "cbbdd4050b2b4ff1bd3428301064fa60"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "7a97f7564f1c6bcc4aa6a13b88ad0ffa"
server_key_iv: "7229cb6047ca56aa202d2de242d54d8f"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.015257 18412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:20.016022 18427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.016208 18412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:20.016294 18412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "cbbdd4050b2b4ff1bd3428301064fa60"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "7a97f7564f1c6bcc4aa6a13b88ad0ffa"
server_key_iv: "7229cb6047ca56aa202d2de242d54d8f"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.016367 18412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:20.029716 18412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:20.031642 18412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:20.031790 18412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:20.035768 18412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39771
I20260924 19:35:20.035815 18479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39771 every 8 connection(s)
I20260924 19:35:20.036140 18412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:20.036984 18480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:20.039198 18480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60: Bootstrap starting.
I20260924 19:35:20.039732 18480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:20.039988 18480 log.cc:824] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:20.040741 18480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60: No bootstrap required, opened a new log
I20260924 19:35:20.042424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 18412
I20260924 19:35:20.042541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:20.042461 18480 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd4050b2b4ff1bd3428301064fa60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } }
I20260924 19:35:20.042631 18480 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:20.042673 18480 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbbdd4050b2b4ff1bd3428301064fa60, State: Initialized, Role: FOLLOWER
I20260924 19:35:20.042747 392 external_mini_cluster.cc:1428] Setting key 50bddd7c653641e6608c8b11a28725d0
I20260924 19:35:20.042815 18480 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd4050b2b4ff1bd3428301064fa60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } }
I20260924 19:35:20.042896 18480 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:20.042958 18480 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:20.042996 18480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:20.043861 18480 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd4050b2b4ff1bd3428301064fa60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } }
I20260924 19:35:20.043951 18480 leader_election.cc:302] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbbdd4050b2b4ff1bd3428301064fa60; no voters:
I20260924 19:35:20.044095 18480 leader_election.cc:288] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:20.044129 18485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:20.044271 18485 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [term 1 LEADER]: Becoming Leader. State: Replica: cbbdd4050b2b4ff1bd3428301064fa60, State: Running, Role: LEADER
I20260924 19:35:20.044353 18480 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:20.044404 18485 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd4050b2b4ff1bd3428301064fa60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } }
I20260924 19:35:20.044925 18487 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbbdd4050b2b4ff1bd3428301064fa60. Latest consensus state: current_term: 1 leader_uuid: "cbbdd4050b2b4ff1bd3428301064fa60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd4050b2b4ff1bd3428301064fa60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } } }
I20260924 19:35:20.045092 18487 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:20.045490 18486 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbbdd4050b2b4ff1bd3428301064fa60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbbdd4050b2b4ff1bd3428301064fa60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39771 } } }
I20260924 19:35:20.045562 18486 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:20.045475 18495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:20.046093 18495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:20.047629 18495 catalog_manager.cc:1347] Generated new cluster ID: 356ad96db2524afc82c9b25aef92ef36
I20260924 19:35:20.047676 18495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:35:20 dist-test-slave-m8jb krb5kdc[18399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:20.054481 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.043204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.98.62:39771)
0924 19:35:20.043389 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.043399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.043442 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:35:20.043555 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.043560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.043600 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.043726 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:20.044473 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.045883 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.046878 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.047395 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.047488 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.051138 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:20.051143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:20.051143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:20.051154 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:20.052940 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.053316 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.053319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.053319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.053423 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.053621 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.053622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.053622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.053898 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:20.054043 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.054140 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.054267 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
I20260924 19:35:20.061328 18495 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:20.061805 18495 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:20.073796 18495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60: Generated new TSK 0
I20260924 19:35:20.073994 18495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:35:20.103118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:20.182427 18506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:20.182619 18506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:20.182641 18506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:20.183712 18506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:20.183744 18506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:20.183768 18506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:35:20.185191 18506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:20.185377 18506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:20.185544 18506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:20.190346 18514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.190335 18511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.190372 18512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.190526 18506 server_base.cc:1034] running on GCE node
I20260924 19:35:20.190726 18506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:20.190968 18506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:20.192164 18506 hybrid_clock.cc:648] HybridClock initialized: now 1790278520192114 us; error 27 us; skew 500 ppm
Sep 24 19:35:20 dist-test-slave-m8jb krb5kdc[18399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:20.194665 18506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:35:20.195078 18506 webserver.cc:460] Webserver started at http://127.0.98.1:40675/ using document root <none> and password file <none>
I20260924 19:35:20.195295 18506 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:20.195343 18506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:20.195457 18506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:20.196514 18506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "71754b228d194f80988496515945ba7b"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "eb903390ddbe56e8e1cf3bb28c44d8d0"
server_key_iv: "935b298a60af3f804203f567eb695165"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.196883 18506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71754b228d194f80988496515945ba7b"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "eb903390ddbe56e8e1cf3bb28c44d8d0"
server_key_iv: "935b298a60af3f804203f567eb695165"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.198187 18506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:20.198978 18521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.199208 18506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:20.199301 18506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "71754b228d194f80988496515945ba7b"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "eb903390ddbe56e8e1cf3bb28c44d8d0"
server_key_iv: "935b298a60af3f804203f567eb695165"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.199366 18506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:20.222684 18506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:20.224865 18506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:20.225004 18506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:20.225226 18506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:20.225526 18506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:20.225559 18506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.225589 18506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:20.225606 18506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.231076 18506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44125
I20260924 19:35:20.231125 18634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44125 every 8 connection(s)
I20260924 19:35:20.231443 18506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:35:20 dist-test-slave-m8jb krb5kdc[18399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:20.238126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 18506
I20260924 19:35:20.238255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:35:20.238451 392 external_mini_cluster.cc:1428] Setting key c1ba19baf7947cc2cbe51198a66ef2fa
I20260924 19:35:20.241135 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.232216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56343 (local address 127.0.98.62:39771)
0924 19:35:20.232286 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.232288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.232500 (+ 212us) server_negotiation.cc:407] Connection header received
0924 19:35:20.233452 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.233453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.233464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.233520 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:20.234155 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.234698 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.235708 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.235908 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.235963 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.238177 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:20.238178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:20.238179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:20.238186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:20.239607 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.240039 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.240040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.240041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.240094 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.240435 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.240437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.240437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.240586 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:20.240709 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.240958 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.241048 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:35:20.241215 18637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.232205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39771 (local address 127.0.98.1:56343)
0924 19:35:20.232375 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.232413 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:20.233346 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:20.233583 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:20.233586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:20.233860 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:20.234046 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.234050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.234794 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.234795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.235612 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.235614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.236004 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.236005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.236122 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.236675 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:20.236686 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:20.238087 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:20.239677 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:20.239684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:20.239686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:20.239967 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:20.240150 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:20.240151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:20.240151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:20.240287 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:20.240700 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:20.240706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:20.240839 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 19:35:20.240921 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:35:20.241044 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260924 19:35:20.241755 18635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39771
I20260924 19:35:20.241855 18635 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:20.242033 18635 heartbeater.cc:511] Master 127.0.98.62:39771 requested a full tablet report, sending...
I20260924 19:35:20.242532 18444 ts_manager.cc:194] Registered new tserver with Master: 71754b228d194f80988496515945ba7b (127.0.98.1:44125)
I20260924 19:35:20.243136 18444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56343
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:35:20.263875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:20.342622 18640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:20.342806 18640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:20.342828 18640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:20.343940 18640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:20.343978 18640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:20.344002 18640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:35:20.345539 18640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:20.345747 18640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:20.345947 18640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:20.350644 18645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.350793 18640 server_base.cc:1034] running on GCE node
W20260924 19:35:20.350649 18646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.350746 18648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.351065 18640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:20.351251 18640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:20.352391 18640 hybrid_clock.cc:648] HybridClock initialized: now 1790278520352378 us; error 27 us; skew 500 ppm
Sep 24 19:35:20 dist-test-slave-m8jb krb5kdc[18399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:20.355012 18640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:35:20.355374 18640 webserver.cc:460] Webserver started at http://127.0.98.2:34669/ using document root <none> and password file <none>
I20260924 19:35:20.355554 18640 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:20.355592 18640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:20.355674 18640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:20.356616 18640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "6ce6f1878e874e0d830f4abc464e7b26"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "ec6ad14887c4e56cecde35732a9ac6ad"
server_key_iv: "d4e13e7450bdc7a95f0e1aba07362ed1"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.356951 18640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6ce6f1878e874e0d830f4abc464e7b26"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "ec6ad14887c4e56cecde35732a9ac6ad"
server_key_iv: "d4e13e7450bdc7a95f0e1aba07362ed1"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.358103 18640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:35:20.358721 18655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.358884 18640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:20.358959 18640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "6ce6f1878e874e0d830f4abc464e7b26"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "ec6ad14887c4e56cecde35732a9ac6ad"
server_key_iv: "d4e13e7450bdc7a95f0e1aba07362ed1"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.359017 18640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:20.368247 18640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:20.370168 18640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:20.370301 18640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:20.370496 18640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:20.370790 18640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:20.370820 18640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.370848 18640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:20.370872 18640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.376178 18640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35725
I20260924 19:35:20.376240 18768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35725 every 8 connection(s)
I20260924 19:35:20.376533 18640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:35:20.378229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 18640
I20260924 19:35:20.378362 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:35:20.378563 392 external_mini_cluster.cc:1428] Setting key c640fb62adeecf46c6f41f5900b0ec87
Sep 24 19:35:20 dist-test-slave-m8jb krb5kdc[18399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:35:20.386472 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.377283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52975 (local address 127.0.98.62:39771)
0924 19:35:20.377328 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.377329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.377504 (+ 175us) server_negotiation.cc:407] Connection header received
0924 19:35:20.378484 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.378485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.378501 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.378575 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:20.379423 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.380118 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.381623 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.381863 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.381920 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.384014 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:20.384015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:20.384016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:20.384023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:20.385250 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.385609 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.385610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.385610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.385660 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.385871 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.385872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.385872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.385979 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:20.386035 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.386289 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.386375 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:35:20.386638 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.377240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39771 (local address 127.0.98.2:52975)
0924 19:35:20.377389 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.377429 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:20.378318 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:20.378650 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:20.378655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:20.379032 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:20.379316 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.379322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.380273 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.380274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.381546 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.381549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.381940 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.381941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.382067 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.382607 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:20.382621 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:20.383925 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:20.385329 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:20.385336 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:20.385338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:20.385543 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:20.385712 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:20.385713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:20.385713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:20.385817 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:20.386031 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:20.386037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:20.386173 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 19:35:20.386260 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:35:20.386428 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260924 19:35:20.387102 18769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39771
I20260924 19:35:20.387199 18769 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:20.387375 18769 heartbeater.cc:511] Master 127.0.98.62:39771 requested a full tablet report, sending...
I20260924 19:35:20.387720 18444 ts_manager.cc:194] Registered new tserver with Master: 6ce6f1878e874e0d830f4abc464e7b26 (127.0.98.2:35725)
I20260924 19:35:20.388154 18444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:35:20.403088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:20.482759 18774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:20.482962 18774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:20.482985 18774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:20.484126 18774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:20.484167 18774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:20.484201 18774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:35:20.485702 18774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:20.485934 18774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:20.486131 18774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:20.490921 18782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.491003 18780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.490957 18779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.491099 18774 server_base.cc:1034] running on GCE node
I20260924 19:35:20.491322 18774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:20.491536 18774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:20.492712 18774 hybrid_clock.cc:648] HybridClock initialized: now 1790278520492696 us; error 28 us; skew 500 ppm
Sep 24 19:35:20 dist-test-slave-m8jb krb5kdc[18399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:20.495347 18774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:35:20.495743 18774 webserver.cc:460] Webserver started at http://127.0.98.3:38597/ using document root <none> and password file <none>
I20260924 19:35:20.495949 18774 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:20.495992 18774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:20.496119 18774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:20.497262 18774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "73aef5e402df4f31ac6fc3efa930fe17"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "e1749b7f20d2dafd65e831ef603be6d9"
server_key_iv: "6b6d87c22a5b247543af62c11eae40f6"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.497586 18774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73aef5e402df4f31ac6fc3efa930fe17"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "e1749b7f20d2dafd65e831ef603be6d9"
server_key_iv: "6b6d87c22a5b247543af62c11eae40f6"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.498891 18774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:20.499684 18789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.499943 18774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:20.500032 18774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "73aef5e402df4f31ac6fc3efa930fe17"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "e1749b7f20d2dafd65e831ef603be6d9"
server_key_iv: "6b6d87c22a5b247543af62c11eae40f6"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.500108 18774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:20.544693 18774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:20.546940 18774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:20.547061 18774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:20.547291 18774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:20.547574 18774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:20.547605 18774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.547627 18774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:20.547650 18774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.552956 18774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44597
I20260924 19:35:20.552994 18902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44597 every 8 connection(s)
I20260924 19:35:20.553308 18774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:35:20.557972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 18774
I20260924 19:35:20.558068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:35:20.558246 392 external_mini_cluster.cc:1428] Setting key cb5eb1550af8f0d74fc21bc54a11ccf3
Sep 24 19:35:20 dist-test-slave-m8jb krb5kdc[18399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278520, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:20.562453 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.554011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37745 (local address 127.0.98.62:39771)
0924 19:35:20.554064 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.554066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.554258 (+ 192us) server_negotiation.cc:407] Connection header received
0924 19:35:20.555159 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.555160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.555171 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.555231 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:20.555873 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.556422 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.557434 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.557625 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.557678 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.559878 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:20.559880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:20.559880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:20.559886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:20.561201 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.561619 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.561619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.561620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.561670 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.561871 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.561871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.561871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.561966 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:20.562010 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.562327 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.562390 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:35:20.562738 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.553957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39771 (local address 127.0.98.3:37745)
0924 19:35:20.554108 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.554147 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:20.555038 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:20.555295 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:20.555299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:20.555549 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:20.555778 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.555783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.556495 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.556495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.557363 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.557365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.557702 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.557703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.557795 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.558338 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:20.558346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:20.559788 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:20.561288 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:20.561294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:20.561295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:20.561556 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:20.561724 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:20.561725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:20.561725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:20.561825 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:20.562040 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:20.562047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:20.562200 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:35:20.562302 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:35:20.562431 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260924 19:35:20.563192 18903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39771
I20260924 19:35:20.563282 18903 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:20.563442 18903 heartbeater.cc:511] Master 127.0.98.62:39771 requested a full tablet report, sending...
I20260924 19:35:20.563781 18444 ts_manager.cc:194] Registered new tserver with Master: 73aef5e402df4f31ac6fc3efa930fe17 (127.0.98.3:44597)
I20260924 19:35:20.564170 18444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37745
I20260924 19:35:20.570039 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:35:20.577160 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.571337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40814 (local address 127.0.98.62:39771)
0924 19:35:20.571410 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.571412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.571478 (+ 66us) server_negotiation.cc:407] Connection header received
0924 19:35:20.571541 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.571541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.571551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.571592 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:20.572091 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.572555 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.573317 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.573500 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.573552 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.574802 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:20.574803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:20.574804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:20.574811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:20.576013 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.576382 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.576383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.576384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.576435 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:20.576620 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:20.576621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:20.576621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:20.576792 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:20.576884 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.577018 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.577075 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:35:20.578729 18444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:35:20.580255 18444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:35:20.586485 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.582317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35725 (local address 127.0.0.1:45522)
0924 19:35:20.582455 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.582491 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:20.582551 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:20.583059 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:20.583060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:20.583062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:20.583115 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.583117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.584372 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.584373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.585474 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.585477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.586180 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.586180 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.586269 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.586281 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:35:20.586345 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:35:20.586408 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":6,"thread_start_us":41,"threads_started":1}
I20260924 19:35:20.586493 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.582463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.0.98.2:35725)
0924 19:35:20.582883 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.582888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.582900 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:35:20.582935 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.582938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.582952 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.583054 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:20.583233 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.584275 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.585558 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.586091 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.586151 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.586192 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.586364 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.586417 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":75,"threads_started":1}
I20260924 19:35:20.586946 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.582332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44597 (local address 127.0.0.1:40878)
0924 19:35:20.582460 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.582491 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:20.582552 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:20.582968 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:20.582972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:20.582975 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:20.583059 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.583061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.584708 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.584709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.585613 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.585616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.586238 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.586239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.586302 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.586306 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:20.586353 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:35:20.586876 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":460,"thread_start_us":90,"threads_started":1}
I20260924 19:35:20.587093 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.582631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.0.98.3:44597)
0924 19:35:20.582802 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.582808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.582819 (+ 11us) server_negotiation.cc:407] Connection header received
0924 19:35:20.582857 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.582859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.582875 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.582955 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:20.583141 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.584269 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.585692 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.586156 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.586227 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.586285 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.586409 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.586459 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260924 19:35:20.587136 18703 tablet_service.cc:1467] Processing CreateTablet for tablet cabc18d9a33a4b84a00af9b26da3af23 (DEFAULT_TABLE table=test-table [id=48b44d96a2be49919d032dfcd2189d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:20.587504 18703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabc18d9a33a4b84a00af9b26da3af23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:20.587581 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.582714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.0.98.1:44125)
0924 19:35:20.582877 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.582881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.582890 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:20.582926 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.582928 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.582941 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.583266 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:20.583349 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.583995 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.586229 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.586948 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.587017 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.587076 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.587456 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.587508 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260924 19:35:20.587743 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.582515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44125 (local address 127.0.0.1:39688)
0924 19:35:20.582720 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.582729 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:20.582771 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:20.583165 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:20.583166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:20.583167 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:20.583197 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.583198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.584083 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.584084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.585474 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.585477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.587109 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.587110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.587171 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.587176 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:35:20.587222 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:35:20.587264 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":27,"thread_start_us":71,"threads_started":1}
I20260924 19:35:20.587720 18837 tablet_service.cc:1467] Processing CreateTablet for tablet cabc18d9a33a4b84a00af9b26da3af23 (DEFAULT_TABLE table=test-table [id=48b44d96a2be49919d032dfcd2189d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:20.588023 18837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabc18d9a33a4b84a00af9b26da3af23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:20.590173 18920 tablet_bootstrap.cc:492] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26: Bootstrap starting.
I20260924 19:35:20.590636 18921 tablet_bootstrap.cc:492] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17: Bootstrap starting.
I20260924 19:35:20.590574 18569 tablet_service.cc:1467] Processing CreateTablet for tablet cabc18d9a33a4b84a00af9b26da3af23 (DEFAULT_TABLE table=test-table [id=48b44d96a2be49919d032dfcd2189d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:20.590828 18920 tablet_bootstrap.cc:654] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:20.590870 18569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabc18d9a33a4b84a00af9b26da3af23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:20.591192 18921 tablet_bootstrap.cc:654] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:20.591393 18920 log.cc:824] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:20.591701 18921 log.cc:824] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:20.592414 18920 tablet_bootstrap.cc:492] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26: No bootstrap required, opened a new log
I20260924 19:35:20.592423 18921 tablet_bootstrap.cc:492] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17: No bootstrap required, opened a new log
I20260924 19:35:20.592486 18921 ts_tablet_manager.cc:1397] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:20.592486 18920 ts_tablet_manager.cc:1397] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:20.593634 18924 tablet_bootstrap.cc:492] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b: Bootstrap starting.
I20260924 19:35:20.594183 18924 tablet_bootstrap.cc:654] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:20.594417 18920 raft_consensus.cc:348] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.594417 18921 raft_consensus.cc:348] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.594599 18921 raft_consensus.cc:374] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:20.594595 18924 log.cc:824] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:20.594599 18920 raft_consensus.cc:374] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:20.594645 18921 raft_consensus.cc:729] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73aef5e402df4f31ac6fc3efa930fe17, State: Initialized, Role: FOLLOWER
I20260924 19:35:20.594645 18920 raft_consensus.cc:729] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce6f1878e874e0d830f4abc464e7b26, State: Initialized, Role: FOLLOWER
I20260924 19:35:20.594758 18920 consensus_queue.cc:260] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.594758 18921 consensus_queue.cc:260] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.595041 18903 heartbeater.cc:503] Master 127.0.98.62:39771 was elected leader, sending a full tablet report...
I20260924 19:35:20.595158 18769 heartbeater.cc:503] Master 127.0.98.62:39771 was elected leader, sending a full tablet report...
I20260924 19:35:20.595266 18920 ts_tablet_manager.cc:1428] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:35:20.595299 18924 tablet_bootstrap.cc:492] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b: No bootstrap required, opened a new log
I20260924 19:35:20.595441 18921 ts_tablet_manager.cc:1428] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:35:20.595747 18924 ts_tablet_manager.cc:1397] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:20.597321 18924 raft_consensus.cc:348] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.597461 18924 raft_consensus.cc:374] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:20.597491 18924 raft_consensus.cc:729] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71754b228d194f80988496515945ba7b, State: Initialized, Role: FOLLOWER
I20260924 19:35:20.597599 18924 consensus_queue.cc:260] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.597819 18924 ts_tablet_manager.cc:1428] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:20.597849 18635 heartbeater.cc:503] Master 127.0.98.62:39771 was elected leader, sending a full tablet report...
W20260924 19:35:20.627131 18770 tablet.cc:2369] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:35:20.652114 18927 raft_consensus.cc:482] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:20.652242 18927 raft_consensus.cc:504] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.652550 18927 leader_election.cc:288] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73aef5e402df4f31ac6fc3efa930fe17 (127.0.98.3:44597), 71754b228d194f80988496515945ba7b (127.0.98.1:44125)
I20260924 19:35:20.656409 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.652883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55609 (local address 127.0.98.3:44597)
0924 19:35:20.652947 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.652949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.652970 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:35:20.652997 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.652998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.653007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.653052 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:20.653256 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.653997 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.655392 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.655998 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.656054 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.656063 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.656290 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.656330 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:35:20.656404 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.653048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56515 (local address 127.0.98.1:44125)
0924 19:35:20.653090 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.653091 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.653238 (+ 147us) server_negotiation.cc:407] Connection header received
0924 19:35:20.653293 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.653294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.653304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.653359 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:20.653488 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.654174 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.655407 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.655963 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.656042 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.656054 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.656290 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.656330 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:35:20.656436 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.652801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44597 (local address 127.0.98.2:55609)
0924 19:35:20.652859 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.652872 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:20.652929 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:20.653082 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:20.653083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:20.653086 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:20.653164 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.653167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.654097 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.654098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.655281 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.655283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.656100 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.656101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.656192 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.656197 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:35:20.656280 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:35:20.656332 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":4}
I20260924 19:35:20.656442 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.652918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44125 (local address 127.0.98.2:56515)
0924 19:35:20.653171 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.653181 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:20.653236 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:20.653374 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:20.653375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:20.653376 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:20.653430 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.653432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.654260 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.654261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.655318 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:20.655320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.656115 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.656116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:20.656192 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.656218 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 19:35:20.656280 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:35:20.656332 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":23,"thread_start_us":124,"threads_started":1}
I20260924 19:35:20.656831 18857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabc18d9a33a4b84a00af9b26da3af23" candidate_uuid: "6ce6f1878e874e0d830f4abc464e7b26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73aef5e402df4f31ac6fc3efa930fe17" is_pre_election: true
I20260924 19:35:20.656916 18589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabc18d9a33a4b84a00af9b26da3af23" candidate_uuid: "6ce6f1878e874e0d830f4abc464e7b26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71754b228d194f80988496515945ba7b" is_pre_election: true
I20260924 19:35:20.656996 18857 raft_consensus.cc:2454] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce6f1878e874e0d830f4abc464e7b26 in term 0.
I20260924 19:35:20.657033 18589 raft_consensus.cc:2454] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce6f1878e874e0d830f4abc464e7b26 in term 0.
I20260924 19:35:20.657210 18657 leader_election.cc:302] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce6f1878e874e0d830f4abc464e7b26, 73aef5e402df4f31ac6fc3efa930fe17; no voters:
I20260924 19:35:20.657428 18927 raft_consensus.cc:2781] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:35:20.657487 18927 raft_consensus.cc:482] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:35:20.657512 18927 raft_consensus.cc:3037] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:20.658202 18927 raft_consensus.cc:504] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.658329 18927 leader_election.cc:288] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [CANDIDATE]: Term 1 election: Requested vote from peers 73aef5e402df4f31ac6fc3efa930fe17 (127.0.98.3:44597), 71754b228d194f80988496515945ba7b (127.0.98.1:44125)
I20260924 19:35:20.658533 18857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabc18d9a33a4b84a00af9b26da3af23" candidate_uuid: "6ce6f1878e874e0d830f4abc464e7b26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73aef5e402df4f31ac6fc3efa930fe17"
I20260924 19:35:20.658574 18589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabc18d9a33a4b84a00af9b26da3af23" candidate_uuid: "6ce6f1878e874e0d830f4abc464e7b26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71754b228d194f80988496515945ba7b"
I20260924 19:35:20.658639 18857 raft_consensus.cc:3037] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:20.658648 18589 raft_consensus.cc:3037] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:20.659478 18857 raft_consensus.cc:2454] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce6f1878e874e0d830f4abc464e7b26 in term 1.
I20260924 19:35:20.659669 18657 leader_election.cc:302] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce6f1878e874e0d830f4abc464e7b26, 73aef5e402df4f31ac6fc3efa930fe17; no voters:
I20260924 19:35:20.659776 18927 raft_consensus.cc:2781] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:20.659977 18927 raft_consensus.cc:686] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [term 1 LEADER]: Becoming Leader. State: Replica: 6ce6f1878e874e0d830f4abc464e7b26, State: Running, Role: LEADER
I20260924 19:35:20.659991 18589 raft_consensus.cc:2454] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce6f1878e874e0d830f4abc464e7b26 in term 1.
I20260924 19:35:20.660084 18927 consensus_queue.cc:237] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } }
I20260924 19:35:20.660980 18444 catalog_manager.cc:5495] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ce6f1878e874e0d830f4abc464e7b26 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6ce6f1878e874e0d830f4abc464e7b26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ce6f1878e874e0d830f4abc464e7b26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 } health_report { overall_health: UNKNOWN } } }
I20260924 19:35:20.680024 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.677793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.0.98.2:35725)
0924 19:35:20.677846 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.677848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.677858 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:35:20.677891 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.677891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.677908 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.677964 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:20.678084 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.678530 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.679315 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.679513 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.679571 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.679753 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:20.679839 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:20.679902 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.679938 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.679948 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:35:20.681751 18589 raft_consensus.cc:1261] T cabc18d9a33a4b84a00af9b26da3af23 P 71754b228d194f80988496515945ba7b [term 1 FOLLOWER]: Refusing update from remote peer 6ce6f1878e874e0d830f4abc464e7b26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:20.681751 18857 raft_consensus.cc:1261] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 1 FOLLOWER]: Refusing update from remote peer 6ce6f1878e874e0d830f4abc464e7b26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:20.682052 18927 consensus_queue.cc:1035] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73aef5e402df4f31ac6fc3efa930fe17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:20.682206 18930 consensus_queue.cc:1035] T cabc18d9a33a4b84a00af9b26da3af23 P 6ce6f1878e874e0d830f4abc464e7b26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71754b228d194f80988496515945ba7b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:20.684791 18934 mvcc.cc:205] Tried to move back new op lower bound from 7332980820710047744 to 7332980820624551936. Current Snapshot: MvccSnapshot[applied={T|T < 7332980820710047744}]
I20260924 19:35:20.686506 18443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:20.686584 18443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:20.687250 18443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbbdd4050b2b4ff1bd3428301064fa60: Sending DeleteTablet for 3 replicas of tablet cabc18d9a33a4b84a00af9b26da3af23
I20260924 19:35:20.687577 18837 tablet_service.cc:1514] Processing DeleteTablet for tablet cabc18d9a33a4b84a00af9b26da3af23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40878
I20260924 19:35:20.687618 18569 tablet_service.cc:1514] Processing DeleteTablet for tablet cabc18d9a33a4b84a00af9b26da3af23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39688
I20260924 19:35:20.687825 18943 tablet_replica.cc:330] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17: stopping tablet replica
I20260924 19:35:20.687929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 18506
I20260924 19:35:20.688162 18703 tablet_service.cc:1514] Processing DeleteTablet for tablet cabc18d9a33a4b84a00af9b26da3af23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45522
W20260924 19:35:20.691946 18429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44125 (error 108)
I20260924 19:35:20.692405 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.692287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44125 (local address 127.0.0.1:39704)
0924 19:35:20.692338 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.692374 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:35:20.692515 18429 catalog_manager.cc:4584] TS 71754b228d194f80988496515945ba7b (127.0.98.1:44125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cabc18d9a33a4b84a00af9b26da3af23: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44125: connect: Connection refused (error 111)
I20260924 19:35:20.692833 18943 raft_consensus.cc:2229] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:20.692965 18943 raft_consensus.cc:2258] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:20.693039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 18640
I20260924 19:35:20.696996 18943 ts_tablet_manager.cc:1905] T cabc18d9a33a4b84a00af9b26da3af23 P 73aef5e402df4f31ac6fc3efa930fe17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:20.697597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 18774
I20260924 19:35:20.697722 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.697613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35725 (local address 127.0.0.1:45540)
0924 19:35:20.697663 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.697693 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:35:20.702235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 18412
I20260924 19:35:20.702278 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.702144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44597 (local address 127.0.0.1:40892)
0924 19:35:20.702217 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:20.702250 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
2026-09-24T19:35:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1867 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T19:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:20Z Disabled control of system clock
I20260924 19:35:20.727764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37489
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:20.806324 18950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:20.806542 18950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:20.806561 18950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:20.807719 18950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:20.807765 18950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:20.807785 18950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:20.807796 18950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:20.807807 18950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:20.809185 18950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:20.809425 18950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:20.809695 18950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:20.814114 18955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.814128 18956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.814327 18950 server_base.cc:1034] running on GCE node
W20260924 19:35:20.814404 18958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.814623 18950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:20.814903 18950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:20.816069 18950 hybrid_clock.cc:648] HybridClock initialized: now 1790278520816051 us; error 27 us; skew 500 ppm
I20260924 19:35:20.817368 18950 webserver.cc:460] Webserver started at http://127.0.98.62:35977/ using document root <none> and password file <none>
I20260924 19:35:20.817603 18950 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:20.817653 18950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:20.817776 18950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:20.818774 18950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "84df076a1dcd4c4e84c1a41111f4e277"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "da95ed784815e8a2df1fa2da669f4a15"
server_key_iv: "f4df0aeffee24c16f7c592041696c540"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.819109 18950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "84df076a1dcd4c4e84c1a41111f4e277"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "da95ed784815e8a2df1fa2da669f4a15"
server_key_iv: "f4df0aeffee24c16f7c592041696c540"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.820333 18950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:20.821064 18964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.821249 18950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.821329 18950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "84df076a1dcd4c4e84c1a41111f4e277"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "da95ed784815e8a2df1fa2da669f4a15"
server_key_iv: "f4df0aeffee24c16f7c592041696c540"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.821395 18950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:20.838002 18950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:20.838318 18950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:20.838472 18950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:20.842245 18950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37489
I20260924 19:35:20.842279 19016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37489 every 8 connection(s)
I20260924 19:35:20.842592 18950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:20.843456 19017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:20.845703 19017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277: Bootstrap starting.
I20260924 19:35:20.846231 19017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:20.846486 19017 log.cc:824] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:20.847082 19017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277: No bootstrap required, opened a new log
I20260924 19:35:20.848613 19017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df076a1dcd4c4e84c1a41111f4e277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37489 } }
I20260924 19:35:20.848789 19017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:20.848814 19017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84df076a1dcd4c4e84c1a41111f4e277, State: Initialized, Role: FOLLOWER
I20260924 19:35:20.848951 19017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df076a1dcd4c4e84c1a41111f4e277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37489 } }
I20260924 19:35:20.849010 19017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:20.849030 19017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:20.849071 19017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:20.849551 19017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df076a1dcd4c4e84c1a41111f4e277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37489 } }
I20260924 19:35:20.849643 19017 leader_election.cc:302] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84df076a1dcd4c4e84c1a41111f4e277; no voters:
I20260924 19:35:20.849792 19017 leader_election.cc:288] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:20.849835 19020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:20.849992 19020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [term 1 LEADER]: Becoming Leader. State: Replica: 84df076a1dcd4c4e84c1a41111f4e277, State: Running, Role: LEADER
I20260924 19:35:20.850021 19017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:20.850085 19020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df076a1dcd4c4e84c1a41111f4e277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37489 } }
I20260924 19:35:20.850476 19022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84df076a1dcd4c4e84c1a41111f4e277. Latest consensus state: current_term: 1 leader_uuid: "84df076a1dcd4c4e84c1a41111f4e277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df076a1dcd4c4e84c1a41111f4e277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37489 } } }
I20260924 19:35:20.850554 19021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84df076a1dcd4c4e84c1a41111f4e277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df076a1dcd4c4e84c1a41111f4e277" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37489 } } }
I20260924 19:35:20.850557 19022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:20.850626 19021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:20.850817 19029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:20.851294 19029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:20.851786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 18950
I20260924 19:35:20.851886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:20.852095 392 external_mini_cluster.cc:1428] Setting key f0bfc752623fc288f53588f04cb5603f
I20260924 19:35:20.852768 19029 catalog_manager.cc:1347] Generated new cluster ID: 67405ce98180429888628f8d7de4249b
I20260924 19:35:20.852818 19029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:20.856675 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:20.852503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.0.98.62:37489)
0924 19:35:20.852682 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:20.852692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:20.852848 (+ 156us) server_negotiation.cc:407] Connection header received
0924 19:35:20.852953 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:20.852959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:20.853018 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:20.853124 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:20.853333 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.854210 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.854940 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:20.855350 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:20.855452 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:20.856036 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:20.856040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:20.856040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:20.856203 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:20.856236 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:20.856247 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:20.856298 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:20.856365 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:35:20.856448 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260924 19:35:20.867038 19029 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:20.867537 19029 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:20.872239 19029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277: Generated new TSK 0
I20260924 19:35:20.872413 19029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:20.880414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37489
--builtin_ntp_servers=127.0.98.20:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:20.959422 19041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:20.959621 19041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:20.959648 19041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:20.960814 19041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:20.960856 19041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:20.960880 19041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:35:20.962195 19041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:20.962412 19041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:20.962677 19041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:20.967852 19047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:20.967841 19049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.968027 19041 server_base.cc:1034] running on GCE node
W20260924 19:35:20.967820 19046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:20.968277 19041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:20.968499 19041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:20.969666 19041 hybrid_clock.cc:648] HybridClock initialized: now 1790278520969644 us; error 32 us; skew 500 ppm
I20260924 19:35:20.970883 19041 webserver.cc:460] Webserver started at http://127.0.98.1:45119/ using document root <none> and password file <none>
I20260924 19:35:20.971122 19041 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:20.971172 19041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:20.971297 19041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:20.972326 19041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "e424b173c87d4630b32290408ca705ae"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "c0f27bd37718ee845fb78ef9485e32c9"
server_key_iv: "1547169c1071339d331ce88f6444a59a"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.972687 19041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e424b173c87d4630b32290408ca705ae"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "c0f27bd37718ee845fb78ef9485e32c9"
server_key_iv: "1547169c1071339d331ce88f6444a59a"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.973984 19041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:20.974777 19055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:20.974932 19041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:20.975013 19041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "e424b173c87d4630b32290408ca705ae"
format_stamp: "Formatted at 2026-09-24 19:35:20 on dist-test-slave-m8jb"
server_key: "c0f27bd37718ee845fb78ef9485e32c9"
server_key_iv: "1547169c1071339d331ce88f6444a59a"
server_key_version: "encryptionkey@0"
I20260924 19:35:20.975124 19041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:21.020445 19041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:21.020792 19041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:21.020924 19041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:21.021166 19041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:21.021485 19041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:21.021518 19041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:21.021548 19041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:21.021569 19041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:21.027143 19041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33385
I20260924 19:35:21.027191 19168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33385 every 8 connection(s)
I20260924 19:35:21.027505 19041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:35:21.032508 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.028238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55331 (local address 127.0.98.62:37489)
0924 19:35:21.028297 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.028299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.028454 (+ 155us) server_negotiation.cc:407] Connection header received
0924 19:35:21.028960 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.028961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.028972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.029033 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:21.029738 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.030244 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.031223 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.031413 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.031465 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.032230 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:21.032231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:21.032231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:21.032236 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:21.032251 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:21.032255 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:21.032295 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.032398 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.032432 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:35:21.032600 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.028156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37489 (local address 127.0.98.1:55331)
0924 19:35:21.028345 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.028387 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:21.028886 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:21.029093 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:21.029097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:21.029105 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:21.029633 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.029637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.030340 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.030341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.031137 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.031140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.031474 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.031475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.031579 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.032061 (+ 482us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:21.032062 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:21.032125 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:21.032148 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:21.032302 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:21.032305 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:21.032307 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:35:21.032354 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:35:21.032430 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260924 19:35:21.033073 19169 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37489
I20260924 19:35:21.033167 19169 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:21.033345 19169 heartbeater.cc:511] Master 127.0.98.62:37489 requested a full tablet report, sending...
I20260924 19:35:21.033882 18981 ts_manager.cc:194] Registered new tserver with Master: e424b173c87d4630b32290408ca705ae (127.0.98.1:33385)
I20260924 19:35:21.034536 18981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55331
I20260924 19:35:21.035416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 19041
I20260924 19:35:21.035509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:35:21.035701 392 external_mini_cluster.cc:1428] Setting key ead851f95d32c4ae759da4d3627418e3
I20260924 19:35:21.036520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37489
--builtin_ntp_servers=127.0.98.20:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:21.114302 19172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:21.114497 19172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:21.114517 19172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:21.115630 19172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:21.115667 19172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:21.115690 19172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:35:21.117017 19172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:21.117235 19172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:21.117499 19172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:21.122217 19177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:21.122357 19172 server_base.cc:1034] running on GCE node
W20260924 19:35:21.122236 19180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:21.122226 19178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:21.122699 19172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:21.122903 19172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:21.124001 19172 hybrid_clock.cc:648] HybridClock initialized: now 1790278521123986 us; error 32 us; skew 500 ppm
I20260924 19:35:21.125254 19172 webserver.cc:460] Webserver started at http://127.0.98.2:39343/ using document root <none> and password file <none>
I20260924 19:35:21.125461 19172 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:21.125509 19172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:21.125633 19172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:21.126703 19172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "04b4999cfcf2430ab5edb2f232ac3dc7"
format_stamp: "Formatted at 2026-09-24 19:35:21 on dist-test-slave-m8jb"
server_key: "5845529c614e50819ca013051d06f5e8"
server_key_iv: "0ac80305f087a97b0f2b53a9de6a0480"
server_key_version: "encryptionkey@0"
I20260924 19:35:21.127020 19172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "04b4999cfcf2430ab5edb2f232ac3dc7"
format_stamp: "Formatted at 2026-09-24 19:35:21 on dist-test-slave-m8jb"
server_key: "5845529c614e50819ca013051d06f5e8"
server_key_iv: "0ac80305f087a97b0f2b53a9de6a0480"
server_key_version: "encryptionkey@0"
I20260924 19:35:21.128276 19172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:21.129091 19186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:21.129297 19172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:35:21.129385 19172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "04b4999cfcf2430ab5edb2f232ac3dc7"
format_stamp: "Formatted at 2026-09-24 19:35:21 on dist-test-slave-m8jb"
server_key: "5845529c614e50819ca013051d06f5e8"
server_key_iv: "0ac80305f087a97b0f2b53a9de6a0480"
server_key_version: "encryptionkey@0"
I20260924 19:35:21.129449 19172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:21.137885 19172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:21.138159 19172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:21.138281 19172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:21.138482 19172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:21.138803 19172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:21.138836 19172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:21.138864 19172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:21.138888 19172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:21.144145 19172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43517
I20260924 19:35:21.144197 19299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43517 every 8 connection(s)
I20260924 19:35:21.144496 19172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:35:21.149418 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.145190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46907 (local address 127.0.98.62:37489)
0924 19:35:21.145234 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.145236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.145459 (+ 223us) server_negotiation.cc:407] Connection header received
0924 19:35:21.145982 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.145983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.145996 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.146046 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:21.146685 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.147174 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.148103 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.148311 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.148365 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.149140 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:21.149142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:21.149142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:21.149147 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:21.149162 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:21.149166 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:21.149218 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.149287 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.149338 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:35:21.149590 19302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.145167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37489 (local address 127.0.98.2:46907)
0924 19:35:21.145351 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.145384 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:21.145881 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:21.146114 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:21.146119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:21.146128 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:21.146603 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.146607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.147262 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.147263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.148025 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.148027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.148378 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.148378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.148501 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.149011 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:21.149012 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:21.149041 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:21.149063 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:21.149224 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:21.149228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:21.149232 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:21.149292 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 19:35:21.149392 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260924 19:35:21.150112 19300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37489
I20260924 19:35:21.150202 19300 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:21.150419 19300 heartbeater.cc:511] Master 127.0.98.62:37489 requested a full tablet report, sending...
I20260924 19:35:21.150806 18981 ts_manager.cc:194] Registered new tserver with Master: 04b4999cfcf2430ab5edb2f232ac3dc7 (127.0.98.2:43517)
I20260924 19:35:21.150883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 19172
I20260924 19:35:21.150990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:35:21.151188 392 external_mini_cluster.cc:1428] Setting key 726f78b64b647aabb68a392f372cdfc2
I20260924 19:35:21.151249 18981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46907
I20260924 19:35:21.152078 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37489
--builtin_ntp_servers=127.0.98.20:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:21.229473 19303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:21.229671 19303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:21.229691 19303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:21.230825 19303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:21.230865 19303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:21.230888 19303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:35:21.232239 19303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:21.232460 19303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:21.232772 19303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:21.237636 19311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:21.237713 19308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:21.237646 19303 server_base.cc:1034] running on GCE node
W20260924 19:35:21.237649 19309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:21.238081 19303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:21.238322 19303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:21.239462 19303 hybrid_clock.cc:648] HybridClock initialized: now 1790278521239452 us; error 30 us; skew 500 ppm
I20260924 19:35:21.240682 19303 webserver.cc:460] Webserver started at http://127.0.98.3:40459/ using document root <none> and password file <none>
I20260924 19:35:21.240906 19303 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:21.240953 19303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:21.241072 19303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:21.242094 19303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "87d9c013018444f18346a7a60797a8a3"
format_stamp: "Formatted at 2026-09-24 19:35:21 on dist-test-slave-m8jb"
server_key: "813df1c1400272e0aad464d96909169a"
server_key_iv: "714bb130ddca501b98e90965cd04dbb3"
server_key_version: "encryptionkey@0"
I20260924 19:35:21.242410 19303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87d9c013018444f18346a7a60797a8a3"
format_stamp: "Formatted at 2026-09-24 19:35:21 on dist-test-slave-m8jb"
server_key: "813df1c1400272e0aad464d96909169a"
server_key_iv: "714bb130ddca501b98e90965cd04dbb3"
server_key_version: "encryptionkey@0"
I20260924 19:35:21.243561 19303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:21.244194 19317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:21.244354 19303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:21.244431 19303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "87d9c013018444f18346a7a60797a8a3"
format_stamp: "Formatted at 2026-09-24 19:35:21 on dist-test-slave-m8jb"
server_key: "813df1c1400272e0aad464d96909169a"
server_key_iv: "714bb130ddca501b98e90965cd04dbb3"
server_key_version: "encryptionkey@0"
I20260924 19:35:21.244495 19303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:21.260739 19303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:21.261034 19303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:21.261157 19303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:21.261364 19303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:21.261672 19303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:21.261704 19303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:21.261742 19303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:21.261767 19303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:21.267215 19303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41669
I20260924 19:35:21.267277 19430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41669 every 8 connection(s)
I20260924 19:35:21.267566 19303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:35:21.272436 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.268208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38165 (local address 127.0.98.62:37489)
0924 19:35:21.268264 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.268267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.268488 (+ 221us) server_negotiation.cc:407] Connection header received
0924 19:35:21.268965 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.268966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.268976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.269026 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:21.269630 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.270115 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.271041 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.271237 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.271324 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.272153 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:21.272155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:21.272155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:21.272160 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:21.272177 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:21.272183 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:21.272240 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.272346 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.272374 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:35:21.272543 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.268206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37489 (local address 127.0.98.3:38165)
0924 19:35:21.268374 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.268409 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:21.268886 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:21.269077 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:21.269080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:21.269087 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:21.269568 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.269572 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.270183 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.270184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.270953 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.270955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.271340 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.271341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.271462 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.271954 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:21.271955 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:21.271982 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:21.272004 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:21.272246 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:21.272249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:21.272251 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:35:21.272299 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 19:35:21.272378 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260924 19:35:21.273041 19431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37489
I20260924 19:35:21.273133 19431 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:21.273303 19431 heartbeater.cc:511] Master 127.0.98.62:37489 requested a full tablet report, sending...
I20260924 19:35:21.273645 18981 ts_manager.cc:194] Registered new tserver with Master: 87d9c013018444f18346a7a60797a8a3 (127.0.98.3:41669)
I20260924 19:35:21.274000 18981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38165
I20260924 19:35:21.276419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 19303
I20260924 19:35:21.276525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:35:21.276726 392 external_mini_cluster.cc:1428] Setting key ab17dbeb6a2858ca80fe4ef343233cb0
I20260924 19:35:21.277513 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:35:21.281600 19040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.278838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.0.98.62:37489)
0924 19:35:21.278892 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.278894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.279010 (+ 116us) server_negotiation.cc:407] Connection header received
0924 19:35:21.279045 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.279046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.279057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.279101 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:21.279303 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.279774 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.280495 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.280696 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.280748 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.281371 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:21.281372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:21.281373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:21.281376 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:21.281389 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:21.281393 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:21.281462 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.281512 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.281522 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:35:21.282871 18981 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:35:21.284343 18981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:35:21.290794 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.286506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.0.98.3:41669)
0924 19:35:21.286670 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.286675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.286746 (+ 71us) server_negotiation.cc:407] Connection header received
0924 19:35:21.286809 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.286811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.286822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.286895 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:21.287022 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.288119 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.289719 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.290429 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.290487 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.290571 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.290672 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.290711 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260924 19:35:21.290820 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.286246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41669 (local address 127.0.0.1:48646)
0924 19:35:21.286684 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.286695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:21.286756 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:21.286897 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:21.286898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:21.286900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:21.286966 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.286968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.288227 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.288228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.289630 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.289633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.290497 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.290498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.290575 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.290597 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 19:35:21.290659 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:35:21.290711 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":36,"thread_start_us":90,"threads_started":1}
I20260924 19:35:21.290844 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.286226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43517 (local address 127.0.0.1:43262)
0924 19:35:21.286391 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.286434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:21.286494 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:21.286887 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:21.286890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:21.286892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:21.286967 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.286969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.288241 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.288242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.289752 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.289754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.290573 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.290574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.290644 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.290648 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:21.290702 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 19:35:21.290743 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":42,"thread_start_us":69,"threads_started":1}
I20260924 19:35:21.291155 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.286336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.98.2:43517)
0924 19:35:21.286726 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.286731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.286740 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:21.286779 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.286781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.286797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.286865 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:21.287053 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.288164 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.289830 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.290354 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.290429 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.290520 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.291030 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.291075 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":82,"threads_started":1}
I20260924 19:35:21.291491 19365 tablet_service.cc:1467] Processing CreateTablet for tablet 8243de6a02e440ea89cb696d2933f6f7 (DEFAULT_TABLE table=test-table [id=a829b2bb16704dea848b107a346403d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:21.291828 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.286425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33385 (local address 127.0.0.1:41066)
0924 19:35:21.286953 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.286962 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:21.287016 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:21.288356 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:21.288357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:21.288359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:21.288397 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.288399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.289360 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.289361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.290302 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.290304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.291609 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.291610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.291675 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.291678 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:21.291723 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:35:21.291762 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":80,"threads_started":1}
I20260924 19:35:21.291868 19365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8243de6a02e440ea89cb696d2933f6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:21.291805 19234 tablet_service.cc:1467] Processing CreateTablet for tablet 8243de6a02e440ea89cb696d2933f6f7 (DEFAULT_TABLE table=test-table [id=a829b2bb16704dea848b107a346403d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:21.292124 19234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8243de6a02e440ea89cb696d2933f6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:21.292122 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.286526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.0.98.1:33385)
0924 19:35:21.286687 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.286690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.288215 (+ 1525us) server_negotiation.cc:407] Connection header received
0924 19:35:21.288257 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.288260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.288275 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.288343 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:21.288450 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.289284 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.290395 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.291522 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.291597 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.291752 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.291777 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.291826 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260924 19:35:21.292696 19103 tablet_service.cc:1467] Processing CreateTablet for tablet 8243de6a02e440ea89cb696d2933f6f7 (DEFAULT_TABLE table=test-table [id=a829b2bb16704dea848b107a346403d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:21.292985 19103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8243de6a02e440ea89cb696d2933f6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:21.295161 19445 tablet_bootstrap.cc:492] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Bootstrap starting.
I20260924 19:35:21.295385 19446 tablet_bootstrap.cc:492] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Bootstrap starting.
I20260924 19:35:21.295598 19447 tablet_bootstrap.cc:492] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae: Bootstrap starting.
I20260924 19:35:21.295935 19446 tablet_bootstrap.cc:654] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:21.296051 19445 tablet_bootstrap.cc:654] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:21.296187 19447 tablet_bootstrap.cc:654] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:21.296218 19446 log.cc:824] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:21.296373 19445 log.cc:824] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:21.296489 19447 log.cc:824] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:21.296921 19446 tablet_bootstrap.cc:492] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: No bootstrap required, opened a new log
I20260924 19:35:21.296991 19446 ts_tablet_manager.cc:1397] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:21.297091 19445 tablet_bootstrap.cc:492] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: No bootstrap required, opened a new log
I20260924 19:35:21.297153 19445 ts_tablet_manager.cc:1397] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:21.297297 19447 tablet_bootstrap.cc:492] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae: No bootstrap required, opened a new log
I20260924 19:35:21.297384 19447 ts_tablet_manager.cc:1397] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:35:21.299154 19445 raft_consensus.cc:348] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.299291 19445 raft_consensus.cc:374] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:21.299316 19445 raft_consensus.cc:729] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04b4999cfcf2430ab5edb2f232ac3dc7, State: Initialized, Role: FOLLOWER
I20260924 19:35:21.299435 19445 consensus_queue.cc:260] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.299475 19446 raft_consensus.cc:348] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.299633 19446 raft_consensus.cc:374] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:21.299669 19446 raft_consensus.cc:729] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d9c013018444f18346a7a60797a8a3, State: Initialized, Role: FOLLOWER
I20260924 19:35:21.299724 19445 ts_tablet_manager.cc:1428] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:35:21.299765 19300 heartbeater.cc:503] Master 127.0.98.62:37489 was elected leader, sending a full tablet report...
I20260924 19:35:21.299793 19446 consensus_queue.cc:260] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.300019 19446 ts_tablet_manager.cc:1428] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:35:21.300148 19431 heartbeater.cc:503] Master 127.0.98.62:37489 was elected leader, sending a full tablet report...
I20260924 19:35:21.300048 19447 raft_consensus.cc:348] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.300186 19447 raft_consensus.cc:374] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:21.300226 19447 raft_consensus.cc:729] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e424b173c87d4630b32290408ca705ae, State: Initialized, Role: FOLLOWER
I20260924 19:35:21.300330 19447 consensus_queue.cc:260] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.300554 19447 ts_tablet_manager.cc:1428] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:35:21.300706 19169 heartbeater.cc:503] Master 127.0.98.62:37489 was elected leader, sending a full tablet report...
W20260924 19:35:21.395066 19301 tablet.cc:2369] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:35:21.486910 19452 raft_consensus.cc:482] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:21.487063 19452 raft_consensus.cc:504] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.487452 19452 leader_election.cc:288] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04b4999cfcf2430ab5edb2f232ac3dc7 (127.0.98.2:43517), e424b173c87d4630b32290408ca705ae (127.0.98.1:33385)
I20260924 19:35:21.490608 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.487873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39633 (local address 127.0.98.2:43517)
0924 19:35:21.487919 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.487921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.487929 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:35:21.487957 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.487958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.487967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.488013 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:21.488145 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.488651 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.489753 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.490266 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.490337 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.490348 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.490511 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.490551 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:35:21.490608 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.487643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43517 (local address 127.0.98.3:39633)
0924 19:35:21.487711 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.487721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:21.487780 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:21.488016 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:21.488017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:21.488018 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:21.488087 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.488089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.488724 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.488725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.489664 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.489666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.490364 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.490365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.490438 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.490442 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:21.490489 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:35:21.490533 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 19:35:21.490783 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.487813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41115 (local address 127.0.98.1:33385)
0924 19:35:21.487858 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.487860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.487932 (+ 72us) server_negotiation.cc:407] Connection header received
0924 19:35:21.487989 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.487990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.488002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.488073 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:21.488283 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.488969 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.489971 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.490431 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.490469 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.490479 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.490682 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.490723 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:35:21.490947 19254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8243de6a02e440ea89cb696d2933f6f7" candidate_uuid: "87d9c013018444f18346a7a60797a8a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" is_pre_election: true
I20260924 19:35:21.490999 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.487742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33385 (local address 127.0.98.3:41115)
0924 19:35:21.487881 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.487889 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:21.487939 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:21.488125 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:21.488126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:21.488128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:21.488203 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.488205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.489051 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.489052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.489863 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:21.489866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.490527 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.490528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:21.490586 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.490589 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:21.490890 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 19:35:21.490929 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":9,"thread_start_us":79,"threads_started":1}
I20260924 19:35:21.491092 19254 raft_consensus.cc:2454] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d9c013018444f18346a7a60797a8a3 in term 0.
I20260924 19:35:21.491325 19321 leader_election.cc:302] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04b4999cfcf2430ab5edb2f232ac3dc7, 87d9c013018444f18346a7a60797a8a3; no voters:
I20260924 19:35:21.491369 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8243de6a02e440ea89cb696d2933f6f7" candidate_uuid: "87d9c013018444f18346a7a60797a8a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e424b173c87d4630b32290408ca705ae" is_pre_election: true
I20260924 19:35:21.491470 19452 raft_consensus.cc:2781] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:35:21.491523 19452 raft_consensus.cc:482] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:35:21.491524 19123 raft_consensus.cc:2454] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d9c013018444f18346a7a60797a8a3 in term 0.
I20260924 19:35:21.491550 19452 raft_consensus.cc:3037] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:21.492498 19452 raft_consensus.cc:504] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.492659 19452 leader_election.cc:288] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [CANDIDATE]: Term 1 election: Requested vote from peers 04b4999cfcf2430ab5edb2f232ac3dc7 (127.0.98.2:43517), e424b173c87d4630b32290408ca705ae (127.0.98.1:33385)
I20260924 19:35:21.492827 19123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8243de6a02e440ea89cb696d2933f6f7" candidate_uuid: "87d9c013018444f18346a7a60797a8a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e424b173c87d4630b32290408ca705ae"
I20260924 19:35:21.492843 19254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8243de6a02e440ea89cb696d2933f6f7" candidate_uuid: "87d9c013018444f18346a7a60797a8a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7"
I20260924 19:35:21.492901 19123 raft_consensus.cc:3037] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:21.492916 19254 raft_consensus.cc:3037] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:21.493600 19123 raft_consensus.cc:2454] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d9c013018444f18346a7a60797a8a3 in term 1.
I20260924 19:35:21.493741 19321 leader_election.cc:302] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d9c013018444f18346a7a60797a8a3, e424b173c87d4630b32290408ca705ae; no voters:
I20260924 19:35:21.493808 19254 raft_consensus.cc:2454] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d9c013018444f18346a7a60797a8a3 in term 1.
I20260924 19:35:21.493860 19452 raft_consensus.cc:2781] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:21.494000 19452 raft_consensus.cc:686] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 1 LEADER]: Becoming Leader. State: Replica: 87d9c013018444f18346a7a60797a8a3, State: Running, Role: LEADER
I20260924 19:35:21.494091 19452 consensus_queue.cc:237] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } }
I20260924 19:35:21.495083 18980 catalog_manager.cc:5495] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87d9c013018444f18346a7a60797a8a3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "87d9c013018444f18346a7a60797a8a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87d9c013018444f18346a7a60797a8a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41669 } health_report { overall_health: HEALTHY } } }
W20260924 19:35:21.518081 19432 tablet.cc:2369] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:35:21.520458 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.518158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48650 (local address 127.0.98.3:41669)
0924 19:35:21.518210 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:21.518212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:21.518257 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:35:21.518287 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:21.518288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:21.518311 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:21.518376 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:21.518490 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.518972 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.519722 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:21.519924 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:21.519976 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:21.520132 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:21.520227 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:21.520309 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:21.520356 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:35:21.520364 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 19:35:21.522166 19254 raft_consensus.cc:1261] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 1 FOLLOWER]: Refusing update from remote peer 87d9c013018444f18346a7a60797a8a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:21.522166 19123 raft_consensus.cc:1261] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 1 FOLLOWER]: Refusing update from remote peer 87d9c013018444f18346a7a60797a8a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:21.522490 19455 consensus_queue.cc:1035] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e424b173c87d4630b32290408ca705ae" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:21.522608 19452 consensus_queue.cc:1035] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04b4999cfcf2430ab5edb2f232ac3dc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:21.526086 19459 mvcc.cc:205] Tried to move back new op lower bound from 7332980824152236032 to 7332980824040882176. Current Snapshot: MvccSnapshot[applied={T|T < 7332980824152236032}]
I20260924 19:35:21.527403 18980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:21.527480 18980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:21.528134 18980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84df076a1dcd4c4e84c1a41111f4e277: Sending DeleteTablet for 3 replicas of tablet 8243de6a02e440ea89cb696d2933f6f7
I20260924 19:35:21.528455 19365 tablet_service.cc:1514] Processing DeleteTablet for tablet 8243de6a02e440ea89cb696d2933f6f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:21 UTC) from {username='slave'} at 127.0.0.1:48646
I20260924 19:35:21.528479 19234 tablet_service.cc:1514] Processing DeleteTablet for tablet 8243de6a02e440ea89cb696d2933f6f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:21 UTC) from {username='slave'} at 127.0.0.1:43262
I20260924 19:35:21.528725 19468 tablet_replica.cc:330] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: stopping tablet replica
I20260924 19:35:21.528715 19103 tablet_service.cc:1514] Processing DeleteTablet for tablet 8243de6a02e440ea89cb696d2933f6f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:21 UTC) from {username='slave'} at 127.0.0.1:41066
I20260924 19:35:21.528877 19468 raft_consensus.cc:2229] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:21.528909 19470 tablet_replica.cc:330] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae: stopping tablet replica
I20260924 19:35:21.528992 19470 raft_consensus.cc:2229] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:21.529042 19468 raft_consensus.cc:2258] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:21.529075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 19041
I20260924 19:35:21.529086 19470 raft_consensus.cc:2258] T 8243de6a02e440ea89cb696d2933f6f7 P e424b173c87d4630b32290408ca705ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:21.530344 19468 ts_tablet_manager.cc:1905] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:21.531920 19469 tablet_replica.cc:330] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: stopping tablet replica
I20260924 19:35:21.531972 19468 ts_tablet_manager.cc:1918] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:35:21.532018 19469 raft_consensus.cc:2229] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:35:21.532039 19468 log.cc:1196] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/8243de6a02e440ea89cb696d2933f6f7
I20260924 19:35:21.532102 19469 raft_consensus.cc:2258] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:21.532287 19468 ts_tablet_manager.cc:1939] T 8243de6a02e440ea89cb696d2933f6f7 P 04b4999cfcf2430ab5edb2f232ac3dc7: Deleting consensus metadata
I20260924 19:35:21.532508 19469 ts_tablet_manager.cc:1905] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:21.532693 18968 catalog_manager.cc:4841] TS 04b4999cfcf2430ab5edb2f232ac3dc7 (127.0.98.2:43517): tablet 8243de6a02e440ea89cb696d2933f6f7 (table test-table [id=a829b2bb16704dea848b107a346403d4]) successfully deleted
I20260924 19:35:21.533951 19469 ts_tablet_manager.cc:1918] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:35:21.534018 19469 log.cc:1196] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/8243de6a02e440ea89cb696d2933f6f7
I20260924 19:35:21.534222 19469 ts_tablet_manager.cc:1939] T 8243de6a02e440ea89cb696d2933f6f7 P 87d9c013018444f18346a7a60797a8a3: Deleting consensus metadata
I20260924 19:35:21.534613 18967 catalog_manager.cc:4841] TS 87d9c013018444f18346a7a60797a8a3 (127.0.98.3:41669): tablet 8243de6a02e440ea89cb696d2933f6f7 (table test-table [id=a829b2bb16704dea848b107a346403d4]) successfully deleted
W20260924 19:35:21.535384 18968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33385 (error 108)
I20260924 19:35:21.535586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 19172
I20260924 19:35:21.535799 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:21.535684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33385 (local address 127.0.0.1:41072)
0924 19:35:21.535730 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:21.535766 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 19:35:21.536116 18968 catalog_manager.cc:4584] TS e424b173c87d4630b32290408ca705ae (127.0.98.1:33385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8243de6a02e440ea89cb696d2933f6f7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33385: connect: Connection refused (error 111)
I20260924 19:35:21.540113 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 19303
I20260924 19:35:21.544397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 18950
2026-09-24T19:35:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (842 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:21 dist-test-slave-m8jb krb5kdc[19474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:21 dist-test-slave-m8jb krb5kdc[19474](info): set up 2 sockets
Sep 24 19:35:21 dist-test-slave-m8jb krb5kdc[19474](info): commencing operation
krb5kdc: starting...
W20260924 19:35:22.484319 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:22 dist-test-slave-m8jb krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:22.599351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:22.676826 19487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:22.677035 19487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:22.677064 19487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:22.678217 19487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:22.678258 19487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:22.678275 19487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:22.678292 19487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:22.678308 19487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:22.679740 19487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:22.679970 19487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:22.680169 19487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:22.684729 19487 server_base.cc:1034] running on GCE node
W20260924 19:35:22.684715 19495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:22.684736 19492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:22.684703 19493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:22.685173 19487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:22.685436 19487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:22.686591 19487 hybrid_clock.cc:648] HybridClock initialized: now 1790278522686588 us; error 38 us; skew 500 ppm
Sep 24 19:35:22 dist-test-slave-m8jb krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:22.694058 19487 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:22.694438 19487 webserver.cc:460] Webserver started at http://127.0.98.62:38821/ using document root <none> and password file <none>
I20260924 19:35:22.694607 19487 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:22.694645 19487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:22.694729 19487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:22.695720 19487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "a38503179ded417c920f7a9d9f4dda01"
format_stamp: "Formatted at 2026-09-24 19:35:22 on dist-test-slave-m8jb"
server_key: "fbfff3df536383ebee5d4edd1c36f7f6"
server_key_iv: "3dc9ec3094b0d5f2bc0a4d648c812dae"
server_key_version: "encryptionkey@0"
I20260924 19:35:22.696007 19487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "a38503179ded417c920f7a9d9f4dda01"
format_stamp: "Formatted at 2026-09-24 19:35:22 on dist-test-slave-m8jb"
server_key: "fbfff3df536383ebee5d4edd1c36f7f6"
server_key_iv: "3dc9ec3094b0d5f2bc0a4d648c812dae"
server_key_version: "encryptionkey@0"
I20260924 19:35:22.697184 19487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:22.697801 19502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:22.697940 19487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:22.698022 19487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "a38503179ded417c920f7a9d9f4dda01"
format_stamp: "Formatted at 2026-09-24 19:35:22 on dist-test-slave-m8jb"
server_key: "fbfff3df536383ebee5d4edd1c36f7f6"
server_key_iv: "3dc9ec3094b0d5f2bc0a4d648c812dae"
server_key_version: "encryptionkey@0"
I20260924 19:35:22.698083 19487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:22.705768 19487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:22.707396 19487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:22.707504 19487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:22.711068 19487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46079
I20260924 19:35:22.711098 19554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46079 every 8 connection(s)
I20260924 19:35:22.711392 19487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:22.712175 19555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:22.713665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 19487
I20260924 19:35:22.713768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:22.713934 392 external_mini_cluster.cc:1428] Setting key d1d5d9f57949a9c1c47764f7361cdddc
I20260924 19:35:22.714604 19555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01: Bootstrap starting.
I20260924 19:35:22.715262 19555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:22.715561 19555 log.cc:824] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:22.716244 19555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01: No bootstrap required, opened a new log
I20260924 19:35:22.718425 19555 raft_consensus.cc:348] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38503179ded417c920f7a9d9f4dda01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46079 } }
I20260924 19:35:22.718588 19555 raft_consensus.cc:374] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:22.718639 19555 raft_consensus.cc:729] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38503179ded417c920f7a9d9f4dda01, State: Initialized, Role: FOLLOWER
I20260924 19:35:22.718778 19555 consensus_queue.cc:260] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38503179ded417c920f7a9d9f4dda01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46079 } }
I20260924 19:35:22.718863 19555 raft_consensus.cc:388] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:22.718904 19555 raft_consensus.cc:482] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:22.718940 19555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:22.719595 19555 raft_consensus.cc:504] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38503179ded417c920f7a9d9f4dda01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46079 } }
I20260924 19:35:22.719691 19555 leader_election.cc:302] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a38503179ded417c920f7a9d9f4dda01; no voters:
I20260924 19:35:22.719844 19555 leader_election.cc:288] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:22.719883 19560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:22.719991 19560 raft_consensus.cc:686] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [term 1 LEADER]: Becoming Leader. State: Replica: a38503179ded417c920f7a9d9f4dda01, State: Running, Role: LEADER
I20260924 19:35:22.720069 19560 consensus_queue.cc:237] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38503179ded417c920f7a9d9f4dda01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46079 } }
I20260924 19:35:22.720110 19555 sys_catalog.cc:564] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:22.720399 19560 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a38503179ded417c920f7a9d9f4dda01. Latest consensus state: current_term: 1 leader_uuid: "a38503179ded417c920f7a9d9f4dda01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38503179ded417c920f7a9d9f4dda01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46079 } } }
I20260924 19:35:22.720409 19561 sys_catalog.cc:455] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a38503179ded417c920f7a9d9f4dda01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a38503179ded417c920f7a9d9f4dda01" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46079 } } }
I20260924 19:35:22.720476 19560 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:22.720525 19561 sys_catalog.cc:458] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:22.720731 19566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 19:35:22 dist-test-slave-m8jb krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:22.721397 19566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:22.722920 19566 catalog_manager.cc:1347] Generated new cluster ID: 2a75c56e4ed942149735ef2497ba70be
I20260924 19:35:22.722961 19566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:22.725111 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:22.714449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.0.98.62:46079)
0924 19:35:22.714645 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:22.714654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:22.714685 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:35:22.715214 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:22.715222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:22.715283 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:22.715429 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:22.716031 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:22.717406 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:22.718178 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:22.718781 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:22.718885 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:22.721657 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:22.721661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:22.721662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:22.721675 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:22.723537 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:22.723904 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:22.723906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:22.723907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:22.724013 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:22.724186 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:22.724187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:22.724188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:22.724480 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:22.724688 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:22.724766 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 19:35:22.724906 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
I20260924 19:35:22.733707 19566 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:22.734387 19566 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:22.737681 19566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01: Generated new TSK 0
I20260924 19:35:22.737849 19566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 19:35:22.767112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46079
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:22.846285 19581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:22.846500 19581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:22.846522 19581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:22.847656 19581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:22.847694 19581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:22.847719 19581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:35:22.849181 19581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:22.849400 19581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:22.849623 19581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:22.854386 19586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:22.854395 19587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:22.854444 19589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:22.854701 19581 server_base.cc:1034] running on GCE node
I20260924 19:35:22.854903 19581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:22.855126 19581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:22.856293 19581 hybrid_clock.cc:648] HybridClock initialized: now 1790278522856277 us; error 31 us; skew 500 ppm
Sep 24 19:35:22 dist-test-slave-m8jb krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:22.858747 19581 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 19:35:22.859189 19581 webserver.cc:460] Webserver started at http://127.0.98.1:42879/ using document root <none> and password file <none>
I20260924 19:35:22.859395 19581 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:22.859441 19581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:22.859568 19581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:22.860571 19581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce2633c0cbb6449abd92e80833c9b218"
format_stamp: "Formatted at 2026-09-24 19:35:22 on dist-test-slave-m8jb"
server_key: "9825290787b907033e0d30640674e727"
server_key_iv: "0a966a3ab84f03e811f40c850d09764a"
server_key_version: "encryptionkey@0"
I20260924 19:35:22.860948 19581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce2633c0cbb6449abd92e80833c9b218"
format_stamp: "Formatted at 2026-09-24 19:35:22 on dist-test-slave-m8jb"
server_key: "9825290787b907033e0d30640674e727"
server_key_iv: "0a966a3ab84f03e811f40c850d09764a"
server_key_version: "encryptionkey@0"
I20260924 19:35:22.862185 19581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:22.862891 19596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:22.863023 19581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:22.863129 19581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "ce2633c0cbb6449abd92e80833c9b218"
format_stamp: "Formatted at 2026-09-24 19:35:22 on dist-test-slave-m8jb"
server_key: "9825290787b907033e0d30640674e727"
server_key_iv: "0a966a3ab84f03e811f40c850d09764a"
server_key_version: "encryptionkey@0"
I20260924 19:35:22.863197 19581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:22.873163 19581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:22.875209 19581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:22.875329 19581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:22.875558 19581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:22.875842 19581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:22.875875 19581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:22.875895 19581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:22.875907 19581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:22.881300 19581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39379
I20260924 19:35:22.881356 19709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39379 every 8 connection(s)
I20260924 19:35:22.881660 19581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 19:35:22 dist-test-slave-m8jb krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:22.890596 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:22.882270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35485 (local address 127.0.98.62:46079)
0924 19:35:22.882330 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:22.882332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:22.882527 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:35:22.883489 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:22.883490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:22.883503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:22.883550 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:22.884184 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:22.884755 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:22.885727 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:22.885939 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:22.885994 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:22.888116 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:22.888117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:22.888118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:22.888126 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:22.889355 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:22.889719 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:22.889721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:22.889721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:22.889771 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:22.889982 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:22.889982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:22.889983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:22.890094 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:22.890149 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:22.890404 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 19:35:22.890493 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:35:22.890746 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:22.882252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46079 (local address 127.0.98.1:35485)
0924 19:35:22.882398 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:22.882439 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:22.883374 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:22.883628 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:22.883631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:22.883889 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:22.884084 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:22.884087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:22.884837 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:22.884838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:22.885644 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:22.885646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:22.886009 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:22.886010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:22.886124 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:22.886637 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:22.886646 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:22.888021 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:22.889428 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:22.889435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:22.889437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:22.889652 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:22.889824 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:22.889825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:22.889825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:22.889926 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:22.890147 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:22.890153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:22.890290 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 19:35:22.890378 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:35:22.890533 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260924 19:35:22.891326 19710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46079
I20260924 19:35:22.891443 19710 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:22.891511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 19581
I20260924 19:35:22.891616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:35:22.891685 19710 heartbeater.cc:511] Master 127.0.98.62:46079 requested a full tablet report, sending...
I20260924 19:35:22.891829 392 external_mini_cluster.cc:1428] Setting key b20f032dad932d2914271a4e2c5ecd0d
I20260924 19:35:22.892267 19519 ts_manager.cc:194] Registered new tserver with Master: ce2633c0cbb6449abd92e80833c9b218 (127.0.98.1:39379)
I20260924 19:35:22.893015 19519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35485
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 19:35:22.916893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46079
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:22.998077 19715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:22.998289 19715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:22.998322 19715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:22.999516 19715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:22.999557 19715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:22.999593 19715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:35:23.001137 19715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:23.001363 19715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:23.001573 19715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:23.006450 19721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.006479 19723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:23.006567 19715 server_base.cc:1034] running on GCE node
W20260924 19:35:23.006572 19720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:23.006875 19715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:23.007091 19715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:23.008225 19715 hybrid_clock.cc:648] HybridClock initialized: now 1790278523008214 us; error 29 us; skew 500 ppm
Sep 24 19:35:23 dist-test-slave-m8jb krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:23.010775 19715 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 19:35:23.011276 19715 webserver.cc:460] Webserver started at http://127.0.98.2:43031/ using document root <none> and password file <none>
I20260924 19:35:23.011472 19715 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:23.011518 19715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:23.011646 19715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:23.012679 19715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "ebe68b15aafd435abf6436af7e3fec5e"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "3f6226ca3e6fa4b4ac2eb7a879f06a1a"
server_key_iv: "e45de2d72c4259de130d28d79ee2eeac"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.013010 19715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ebe68b15aafd435abf6436af7e3fec5e"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "3f6226ca3e6fa4b4ac2eb7a879f06a1a"
server_key_iv: "e45de2d72c4259de130d28d79ee2eeac"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.014293 19715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:35:23.015076 19730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.015252 19715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:35:23.015339 19715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "ebe68b15aafd435abf6436af7e3fec5e"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "3f6226ca3e6fa4b4ac2eb7a879f06a1a"
server_key_iv: "e45de2d72c4259de130d28d79ee2eeac"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.015407 19715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:23.040450 19715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:23.042584 19715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:23.042723 19715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:23.042943 19715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:23.043238 19715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:23.043272 19715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.043299 19715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:23.043318 19715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.048651 19715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41225
I20260924 19:35:23.048725 19843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41225 every 8 connection(s)
I20260924 19:35:23.049002 19715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:35:23.051687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 19715
I20260924 19:35:23.051795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:35:23.051968 392 external_mini_cluster.cc:1428] Setting key 15480ce014458e9e86049d8253da4030
Sep 24 19:35:23 dist-test-slave-m8jb krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:23.058727 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.049664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53233 (local address 127.0.98.62:46079)
0924 19:35:23.049740 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.049742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.049883 (+ 141us) server_negotiation.cc:407] Connection header received
0924 19:35:23.050755 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.050756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.050765 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.050802 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:23.051354 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.052081 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.053189 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.053382 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.053435 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.055804 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:23.055805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:23.055805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:23.055812 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:23.057184 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:23.057643 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:23.057644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:23.057645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:23.057702 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:23.058023 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:23.058024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:23.058024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:23.058183 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:23.058247 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.058574 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.058650 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:35:23.058970 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.049635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46079 (local address 127.0.98.2:53233)
0924 19:35:23.049773 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.049808 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.050663 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.050853 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.050856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.051101 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:23.051285 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.051289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.052162 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.052163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.053115 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.053118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.053446 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.053447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.053570 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.054098 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:23.054108 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:23.055698 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:23.057258 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:23.057264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:23.057267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:23.057557 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:23.057780 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:23.057780 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:23.057781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:23.057914 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:23.058258 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:23.058264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:23.058427 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.058560 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.058735 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":81,"threads_started":1}
I20260924 19:35:23.059433 19844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46079
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 19:35:23.059532 19844 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:23.059697 19844 heartbeater.cc:511] Master 127.0.98.62:46079 requested a full tablet report, sending...
I20260924 19:35:23.060053 19519 ts_manager.cc:194] Registered new tserver with Master: ebe68b15aafd435abf6436af7e3fec5e (127.0.98.2:41225)
I20260924 19:35:23.060386 19519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 19:35:23.075759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46079
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:23.153402 19849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:23.153594 19849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:23.153614 19849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:23.154736 19849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:23.154776 19849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:23.154801 19849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:35:23.156225 19849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:23.156424 19849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:23.156620 19849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:23.161270 19855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.161264 19854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.161264 19857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:23.161620 19849 server_base.cc:1034] running on GCE node
I20260924 19:35:23.161801 19849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:23.161981 19849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:23.163110 19849 hybrid_clock.cc:648] HybridClock initialized: now 1790278523163097 us; error 24 us; skew 500 ppm
Sep 24 19:35:23 dist-test-slave-m8jb krb5kdc[19474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:23.165365 19849 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 19:35:23.165781 19849 webserver.cc:460] Webserver started at http://127.0.98.3:39687/ using document root <none> and password file <none>
I20260924 19:35:23.165995 19849 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:23.166054 19849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:23.166177 19849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:23.167162 19849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "a2cdfe21dbc14e80a57d05abc499cf0f"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "401b5818dcb5b3bdaaab14e84e4ec069"
server_key_iv: "df4b8f7907e8e160da8306779dec66f9"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.167469 19849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a2cdfe21dbc14e80a57d05abc499cf0f"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "401b5818dcb5b3bdaaab14e84e4ec069"
server_key_iv: "df4b8f7907e8e160da8306779dec66f9"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.168681 19849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:35:23.169324 19864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.169466 19849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:23.169544 19849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "a2cdfe21dbc14e80a57d05abc499cf0f"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "401b5818dcb5b3bdaaab14e84e4ec069"
server_key_iv: "df4b8f7907e8e160da8306779dec66f9"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.169607 19849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:23.182525 19849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:23.184473 19849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:23.184604 19849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:23.184861 19849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:23.185149 19849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:23.185181 19849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.185210 19849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:23.185227 19849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.190591 19849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43157
I20260924 19:35:23.190637 19977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43157 every 8 connection(s)
I20260924 19:35:23.190925 19849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 19:35:23 dist-test-slave-m8jb krb5kdc[19474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:23.199831 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.191607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43931 (local address 127.0.98.62:46079)
0924 19:35:23.191662 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.191663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.191845 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:35:23.192760 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.192761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.192773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.192835 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:23.193468 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.194031 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.195007 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.195206 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.195268 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.197413 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:23.197414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:23.197415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:23.197421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:23.198691 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:23.199024 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:23.199024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:23.199024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:23.199071 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:23.199277 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:23.199282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:23.199282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:23.199427 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:23.199494 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.199699 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.199764 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 19:35:23.199975 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.191561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46079 (local address 127.0.98.3:43931)
0924 19:35:23.191702 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.191743 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.192607 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.192903 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.192906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.193163 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:23.193371 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.193375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.194111 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.194111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.194915 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.194917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.195291 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.195292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.195392 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.195888 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:23.195896 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:23.197331 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:23.198759 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:23.198764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:23.198766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:23.198963 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:23.199116 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:23.199117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:23.199117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:23.199199 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:23.199487 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:23.199490 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:23.199602 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.199676 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.199787 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260924 19:35:23.200330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 19849
I20260924 19:35:23.200433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:35:23.200552 19978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46079
I20260924 19:35:23.200600 392 external_mini_cluster.cc:1428] Setting key 6a317232f69f999780813ec26464ea43
I20260924 19:35:23.200675 19978 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:23.200868 19978 heartbeater.cc:511] Master 127.0.98.62:46079 requested a full tablet report, sending...
I20260924 19:35:23.201280 19518 ts_manager.cc:194] Registered new tserver with Master: a2cdfe21dbc14e80a57d05abc499cf0f (127.0.98.3:43157)
I20260924 19:35:23.201681 19518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43931
I20260924 19:35:23.212273 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:35:23.219451 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.213703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.0.98.62:46079)
0924 19:35:23.213770 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.213772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.213907 (+ 135us) server_negotiation.cc:407] Connection header received
0924 19:35:23.213932 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.213932 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.213942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.214025 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:23.214557 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.215084 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.215819 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.216055 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.216123 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.217257 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:23.217258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:23.217258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:23.217265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:23.218436 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:23.218736 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:23.218737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:23.218737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:23.218786 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:23.218933 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:23.218934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:23.218934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:23.219079 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:23.219171 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.219294 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.219364 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:35:23.220922 19518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:35:23.222530 19518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:35:23.228253 19986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.224386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43157 (local address 127.0.0.1:35796)
0924 19:35:23.224783 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.224791 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.224826 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.225095 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.225098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.225100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:23.225162 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.225164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.226106 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.226107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.227261 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.227263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.227975 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.227976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.228060 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.228072 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.228128 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.228177 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":4,"thread_start_us":54,"threads_started":1}
I20260924 19:35:23.228264 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.224682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.0.98.3:43157)
0924 19:35:23.224858 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.224863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.224872 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:23.224908 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.224910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.224924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.225000 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:23.225239 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.226020 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.227350 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.227906 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.227963 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.228020 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.228147 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.228186 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260924 19:35:23.228613 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.224385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41225 (local address 127.0.0.1:43754)
0924 19:35:23.224557 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.224594 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.224693 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.225163 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.225164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.225165 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:23.225216 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.225217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.226433 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.226434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.227518 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.227521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.228381 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.228382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.228447 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.228452 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.228509 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.228547 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":34,"thread_start_us":67,"threads_started":1}
I20260924 19:35:23.228672 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.224550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.0.98.1:39379)
0924 19:35:23.224728 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.224734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.224985 (+ 251us) server_negotiation.cc:407] Connection header received
0924 19:35:23.225158 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.225161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.225177 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.225245 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:23.225522 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.226551 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.227510 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.228229 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.228295 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.228347 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.228537 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.228578 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":96,"threads_started":1}
I20260924 19:35:23.228886 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.224386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39379 (local address 127.0.0.1:59340)
0924 19:35:23.224891 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.224901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.225053 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.225386 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.225387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.225389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:23.225432 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.225433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.226637 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.226637 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.227410 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.227412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.228382 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.228383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.228447 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.228452 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.228784 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.228825 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":419,"thread_start_us":36,"threads_started":1}
I20260924 19:35:23.228916 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.224475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43754 (local address 127.0.98.2:41225)
0924 19:35:23.224934 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.224939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.224947 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:35:23.224986 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.224988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.225004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.225076 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:23.225277 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.226332 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.227586 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.228238 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.228295 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.228346 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.228805 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.228849 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":104,"threads_started":1}
I20260924 19:35:23.229107 19912 tablet_service.cc:1467] Processing CreateTablet for tablet ea7aae180c8846a48acbc52293945798 (DEFAULT_TABLE table=test-table [id=bfad4163f4144a70898a6c12f92b87bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:23.229456 19912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7aae180c8846a48acbc52293945798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:23.229415 19778 tablet_service.cc:1467] Processing CreateTablet for tablet ea7aae180c8846a48acbc52293945798 (DEFAULT_TABLE table=test-table [id=bfad4163f4144a70898a6c12f92b87bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:23.229887 19778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7aae180c8846a48acbc52293945798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:23.232146 19992 tablet_bootstrap.cc:492] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: Bootstrap starting.
I20260924 19:35:23.232197 19993 tablet_bootstrap.cc:492] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e: Bootstrap starting.
I20260924 19:35:23.232604 19644 tablet_service.cc:1467] Processing CreateTablet for tablet ea7aae180c8846a48acbc52293945798 (DEFAULT_TABLE table=test-table [id=bfad4163f4144a70898a6c12f92b87bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:23.232858 19993 tablet_bootstrap.cc:654] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:23.232955 19644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea7aae180c8846a48acbc52293945798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:23.233079 19992 tablet_bootstrap.cc:654] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:23.233409 19993 log.cc:824] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:23.233544 19992 log.cc:824] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:23.234277 19992 tablet_bootstrap.cc:492] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: No bootstrap required, opened a new log
I20260924 19:35:23.234345 19992 ts_tablet_manager.cc:1397] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:35:23.235554 19996 tablet_bootstrap.cc:492] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218: Bootstrap starting.
I20260924 19:35:23.235754 19992 raft_consensus.cc:348] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.235932 19992 raft_consensus.cc:374] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:23.235967 19992 raft_consensus.cc:729] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2cdfe21dbc14e80a57d05abc499cf0f, State: Initialized, Role: FOLLOWER
I20260924 19:35:23.236065 19993 tablet_bootstrap.cc:492] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e: No bootstrap required, opened a new log
I20260924 19:35:23.236129 19993 ts_tablet_manager.cc:1397] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 19:35:23.236081 19992 consensus_queue.cc:260] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.236354 19992 ts_tablet_manager.cc:1428] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:35:23.236372 19978 heartbeater.cc:503] Master 127.0.98.62:46079 was elected leader, sending a full tablet report...
I20260924 19:35:23.237077 19996 tablet_bootstrap.cc:654] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:23.237617 19996 log.cc:824] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:23.238090 19993 raft_consensus.cc:348] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.238256 19993 raft_consensus.cc:374] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:23.238291 19993 raft_consensus.cc:729] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe68b15aafd435abf6436af7e3fec5e, State: Initialized, Role: FOLLOWER
I20260924 19:35:23.238312 19996 tablet_bootstrap.cc:492] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218: No bootstrap required, opened a new log
I20260924 19:35:23.238363 19996 ts_tablet_manager.cc:1397] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 19:35:23.238399 19993 consensus_queue.cc:260] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.238680 19844 heartbeater.cc:503] Master 127.0.98.62:46079 was elected leader, sending a full tablet report...
I20260924 19:35:23.238682 19993 ts_tablet_manager.cc:1428] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 19:35:23.239959 19996 raft_consensus.cc:348] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.240089 19996 raft_consensus.cc:374] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:23.240120 19996 raft_consensus.cc:729] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2633c0cbb6449abd92e80833c9b218, State: Initialized, Role: FOLLOWER
I20260924 19:35:23.240228 19996 consensus_queue.cc:260] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.240489 19996 ts_tablet_manager.cc:1428] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:35:23.240492 19710 heartbeater.cc:503] Master 127.0.98.62:46079 was elected leader, sending a full tablet report...
I20260924 19:35:23.288774 19999 raft_consensus.cc:482] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:23.288962 19999 raft_consensus.cc:504] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.289343 19999 leader_election.cc:288] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce2633c0cbb6449abd92e80833c9b218 (127.0.98.1:39379), a2cdfe21dbc14e80a57d05abc499cf0f (127.0.98.3:43157)
I20260924 19:35:23.292260 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.289611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35803 (local address 127.0.98.1:39379)
0924 19:35:23.289681 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.289683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.289733 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:35:23.289808 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.289810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.289823 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.289876 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:23.290042 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.290533 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.291539 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.291925 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.291986 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.291998 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.292153 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.292189 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:35:23.292274 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.289638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56101 (local address 127.0.98.3:43157)
0924 19:35:23.289680 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.289682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.289848 (+ 166us) server_negotiation.cc:407] Connection header received
0924 19:35:23.289886 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.289887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.289898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.289953 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:23.290111 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.290567 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.291558 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.291934 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.292003 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.292013 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.292159 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.292201 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:35:23.292286 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.289531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39379 (local address 127.0.98.2:35803)
0924 19:35:23.289676 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.289688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.289751 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.289888 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.289889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.289891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:23.289974 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.289976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.290612 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.290613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.291446 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.291449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.292003 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.292003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.292086 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.292089 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.292142 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.292187 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1}
I20260924 19:35:23.292294 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.289564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43157 (local address 127.0.98.2:56101)
0924 19:35:23.289758 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.289768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.289831 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.289973 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.289974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.289975 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:23.290045 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.290046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.290647 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.290648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.291456 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.291458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.292011 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.292012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.292089 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.292092 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.292137 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.292184 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":75,"thread_start_us":118,"threads_started":1}
I20260924 19:35:23.292654 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7aae180c8846a48acbc52293945798" candidate_uuid: "ebe68b15aafd435abf6436af7e3fec5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2633c0cbb6449abd92e80833c9b218" is_pre_election: true
I20260924 19:35:23.292821 19664 raft_consensus.cc:2454] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe68b15aafd435abf6436af7e3fec5e in term 0.
I20260924 19:35:23.292905 19932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7aae180c8846a48acbc52293945798" candidate_uuid: "ebe68b15aafd435abf6436af7e3fec5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" is_pre_election: true
I20260924 19:35:23.293033 19731 leader_election.cc:302] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce2633c0cbb6449abd92e80833c9b218, ebe68b15aafd435abf6436af7e3fec5e; no voters:
I20260924 19:35:23.293075 19932 raft_consensus.cc:2454] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebe68b15aafd435abf6436af7e3fec5e in term 0.
I20260924 19:35:23.293185 19999 raft_consensus.cc:2781] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:35:23.293239 19999 raft_consensus.cc:482] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:35:23.293262 19999 raft_consensus.cc:3037] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:23.293887 19999 raft_consensus.cc:504] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.294024 19999 leader_election.cc:288] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [CANDIDATE]: Term 1 election: Requested vote from peers ce2633c0cbb6449abd92e80833c9b218 (127.0.98.1:39379), a2cdfe21dbc14e80a57d05abc499cf0f (127.0.98.3:43157)
I20260924 19:35:23.294198 19932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7aae180c8846a48acbc52293945798" candidate_uuid: "ebe68b15aafd435abf6436af7e3fec5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f"
I20260924 19:35:23.294196 19664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea7aae180c8846a48acbc52293945798" candidate_uuid: "ebe68b15aafd435abf6436af7e3fec5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2633c0cbb6449abd92e80833c9b218"
I20260924 19:35:23.294281 19932 raft_consensus.cc:3037] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:23.294281 19664 raft_consensus.cc:3037] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:23.295148 19664 raft_consensus.cc:2454] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe68b15aafd435abf6436af7e3fec5e in term 1.
I20260924 19:35:23.295148 19932 raft_consensus.cc:2454] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebe68b15aafd435abf6436af7e3fec5e in term 1.
I20260924 19:35:23.295300 19731 leader_election.cc:302] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce2633c0cbb6449abd92e80833c9b218, ebe68b15aafd435abf6436af7e3fec5e; no voters:
I20260924 19:35:23.295393 19999 raft_consensus.cc:2781] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:23.295542 19999 raft_consensus.cc:686] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 1 LEADER]: Becoming Leader. State: Replica: ebe68b15aafd435abf6436af7e3fec5e, State: Running, Role: LEADER
I20260924 19:35:23.295621 19999 consensus_queue.cc:237] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } }
I20260924 19:35:23.296427 19518 catalog_manager.cc:5495] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e reported cstate change: term changed from 0 to 1, leader changed from <none> to ebe68b15aafd435abf6436af7e3fec5e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ebe68b15aafd435abf6436af7e3fec5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe68b15aafd435abf6436af7e3fec5e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 } health_report { overall_health: UNKNOWN } } }
W20260924 19:35:23.299485 19845 tablet.cc:2369] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:35:23.301434 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.299068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.0.98.2:41225)
0924 19:35:23.299117 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.299118 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.299125 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:23.299149 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.299149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.299166 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.299222 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:23.299320 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.299862 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.300720 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.300922 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.300974 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.301108 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:23.301225 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:23.301293 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.301331 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.301349 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 19:35:23.302953 19932 raft_consensus.cc:1261] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 1 FOLLOWER]: Refusing update from remote peer ebe68b15aafd435abf6436af7e3fec5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:23.302992 19664 raft_consensus.cc:1261] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 1 FOLLOWER]: Refusing update from remote peer ebe68b15aafd435abf6436af7e3fec5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:23.303262 20002 consensus_queue.cc:1035] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2cdfe21dbc14e80a57d05abc499cf0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:23.303365 19999 consensus_queue.cc:1035] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce2633c0cbb6449abd92e80833c9b218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:23.305620 20005 mvcc.cc:205] Tried to move back new op lower bound from 7332980831446614016 to 7332980831419527168. Current Snapshot: MvccSnapshot[applied={T|T < 7332980831446614016}]
I20260924 19:35:23.306478 20006 mvcc.cc:205] Tried to move back new op lower bound from 7332980831446614016 to 7332980831419527168. Current Snapshot: MvccSnapshot[applied={T|T < 7332980831446614016}]
I20260924 19:35:23.308136 19519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:23.308213 19519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:23.308818 19519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a38503179ded417c920f7a9d9f4dda01: Sending DeleteTablet for 3 replicas of tablet ea7aae180c8846a48acbc52293945798
I20260924 19:35:23.309085 19778 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7aae180c8846a48acbc52293945798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43754
I20260924 19:35:23.309140 19644 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7aae180c8846a48acbc52293945798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59340
I20260924 19:35:23.309182 19912 tablet_service.cc:1514] Processing DeleteTablet for tablet ea7aae180c8846a48acbc52293945798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35796
I20260924 19:35:23.309325 20015 tablet_replica.cc:330] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218: stopping tablet replica
I20260924 19:35:23.309383 20016 tablet_replica.cc:330] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: stopping tablet replica
I20260924 19:35:23.309413 20015 raft_consensus.cc:2229] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:23.309464 20016 raft_consensus.cc:2229] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:23.309504 20015 raft_consensus.cc:2258] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:23.309538 20016 raft_consensus.cc:2258] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:23.309917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 19581
I20260924 19:35:23.309926 20016 ts_tablet_manager.cc:1905] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:23.309926 20015 ts_tablet_manager.cc:1905] T ea7aae180c8846a48acbc52293945798 P ce2633c0cbb6449abd92e80833c9b218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:23.311533 20016 ts_tablet_manager.cc:1918] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:35:23.311582 20016 log.cc:1196] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/ea7aae180c8846a48acbc52293945798
I20260924 19:35:23.311766 20016 ts_tablet_manager.cc:1939] T ea7aae180c8846a48acbc52293945798 P a2cdfe21dbc14e80a57d05abc499cf0f: Deleting consensus metadata
I20260924 19:35:23.312145 19504 catalog_manager.cc:4841] TS a2cdfe21dbc14e80a57d05abc499cf0f (127.0.98.3:43157): tablet ea7aae180c8846a48acbc52293945798 (table test-table [id=bfad4163f4144a70898a6c12f92b87bb]) successfully deleted
I20260924 19:35:23.316877 20014 tablet_replica.cc:330] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e: stopping tablet replica
I20260924 19:35:23.317005 20014 raft_consensus.cc:2229] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:35:23.317157 20014 raft_consensus.cc:2258] T ea7aae180c8846a48acbc52293945798 P ebe68b15aafd435abf6436af7e3fec5e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:35:23.317200 19503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39379 (error 108)
I20260924 19:35:23.317381 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 19715
I20260924 19:35:23.318367 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.318246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39379 (local address 127.0.0.1:59354)
0924 19:35:23.318301 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.318338 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 19:35:23.318473 19503 catalog_manager.cc:4584] TS ce2633c0cbb6449abd92e80833c9b218 (127.0.98.1:39379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea7aae180c8846a48acbc52293945798: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39379: connect: Connection refused (error 111)
I20260924 19:35:23.321856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 19849
I20260924 19:35:23.321923 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.321812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41225 (local address 127.0.0.1:43774)
0924 19:35:23.321859 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.321888 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:35:23.325410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 19487
2026-09-24T19:35:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1780 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T19:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:23Z Disabled control of system clock
I20260924 19:35:23.350934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44143
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:23.428848 20021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:23.429060 20021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:23.429080 20021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:23.430285 20021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:23.430322 20021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:23.430334 20021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:23.430346 20021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:23.430364 20021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:23.431699 20021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:23.431928 20021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:23.432196 20021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:23.436858 20029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.436913 20027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:23.437021 20021 server_base.cc:1034] running on GCE node
W20260924 19:35:23.436980 20026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:23.437316 20021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:23.437600 20021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:23.438733 20021 hybrid_clock.cc:648] HybridClock initialized: now 1790278523438722 us; error 28 us; skew 500 ppm
I20260924 19:35:23.439916 20021 webserver.cc:460] Webserver started at http://127.0.98.62:38259/ using document root <none> and password file <none>
I20260924 19:35:23.440176 20021 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:23.440225 20021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:23.440333 20021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:23.441527 20021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "7f327b0c1be44917a8dfe74465495834"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "180b59adcd094b14cd352041db2873a4"
server_key_iv: "dec1244cffd809863b3ca43c31d18abf"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.441978 20021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f327b0c1be44917a8dfe74465495834"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "180b59adcd094b14cd352041db2873a4"
server_key_iv: "dec1244cffd809863b3ca43c31d18abf"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.443423 20021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:23.444253 20035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.444439 20021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:23.444525 20021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "7f327b0c1be44917a8dfe74465495834"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "180b59adcd094b14cd352041db2873a4"
server_key_iv: "dec1244cffd809863b3ca43c31d18abf"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.444583 20021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:23.461829 20021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:23.462136 20021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:23.462255 20021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:23.466065 20021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44143
I20260924 19:35:23.466099 20087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44143 every 8 connection(s)
I20260924 19:35:23.466418 20021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:23.467337 20088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:23.469695 20088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834: Bootstrap starting.
I20260924 19:35:23.470242 20088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:23.470489 20088 log.cc:824] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:23.471170 20088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834: No bootstrap required, opened a new log
I20260924 19:35:23.472795 20088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f327b0c1be44917a8dfe74465495834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44143 } }
I20260924 19:35:23.472937 20088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:23.472960 20088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f327b0c1be44917a8dfe74465495834, State: Initialized, Role: FOLLOWER
I20260924 19:35:23.473074 20088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f327b0c1be44917a8dfe74465495834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44143 } }
I20260924 19:35:23.473147 20088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:23.473170 20088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:23.473202 20088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:23.473688 20088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f327b0c1be44917a8dfe74465495834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44143 } }
I20260924 19:35:23.473770 20088 leader_election.cc:302] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f327b0c1be44917a8dfe74465495834; no voters:
I20260924 19:35:23.473896 20088 leader_election.cc:288] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:23.473950 20091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:23.474113 20088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:23.474134 20091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [term 1 LEADER]: Becoming Leader. State: Replica: 7f327b0c1be44917a8dfe74465495834, State: Running, Role: LEADER
I20260924 19:35:23.474226 20091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f327b0c1be44917a8dfe74465495834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44143 } }
I20260924 19:35:23.474602 20093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f327b0c1be44917a8dfe74465495834. Latest consensus state: current_term: 1 leader_uuid: "7f327b0c1be44917a8dfe74465495834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f327b0c1be44917a8dfe74465495834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44143 } } }
I20260924 19:35:23.474690 20093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:23.474691 20092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f327b0c1be44917a8dfe74465495834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f327b0c1be44917a8dfe74465495834" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44143 } } }
I20260924 19:35:23.474785 20092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:23.474987 20100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:23.475210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 20021
I20260924 19:35:23.475311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:23.475500 20100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:23.475510 392 external_mini_cluster.cc:1428] Setting key 32217387e723613ee71f0a6bf102598e
I20260924 19:35:23.476979 20100 catalog_manager.cc:1347] Generated new cluster ID: ee947b0ed20242a18f7b3c062ae8bafb
I20260924 19:35:23.477029 20100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:23.481525 20100 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:23.481698 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.476086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48646 (local address 127.0.98.62:44143)
0924 19:35:23.476348 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.476356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.476389 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:35:23.476498 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.476505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.476572 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.476721 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:23.477059 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.478364 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.479437 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.480070 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.480190 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.480856 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:23.480860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:23.480861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:23.481077 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:23.481121 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:23.481136 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:23.481205 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.481278 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.481354 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":67,"threads_started":1}
I20260924 19:35:23.482108 20100 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:23.486658 20100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834: Generated new TSK 0
I20260924 19:35:23.486845 20100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:23.492414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44143
--builtin_ntp_servers=127.0.98.20:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:23.572234 20112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:23.572428 20112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:23.572448 20112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:23.573691 20112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:23.573740 20112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:23.573786 20112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 19:35:23.575241 20112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:23.575474 20112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:23.575752 20112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:23.581708 20120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.581712 20117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.581744 20118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:23.582108 20112 server_base.cc:1034] running on GCE node
I20260924 19:35:23.582311 20112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:23.582545 20112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:23.583709 20112 hybrid_clock.cc:648] HybridClock initialized: now 1790278523583695 us; error 29 us; skew 500 ppm
I20260924 19:35:23.585129 20112 webserver.cc:460] Webserver started at http://127.0.98.1:34455/ using document root <none> and password file <none>
I20260924 19:35:23.585387 20112 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:23.585444 20112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:23.585569 20112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:23.586691 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/data/instance:
uuid: "c2007cca0bcb43d29711db5253634218"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "59a8e0ef8560dc9666eaea35e08504d6"
server_key_iv: "415c71c0d9dc9adbb87af02c903cefe5"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.587018 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c2007cca0bcb43d29711db5253634218"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "59a8e0ef8560dc9666eaea35e08504d6"
server_key_iv: "415c71c0d9dc9adbb87af02c903cefe5"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.588214 20112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:23.588950 20126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.589143 20112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.589242 20112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/wal
uuid: "c2007cca0bcb43d29711db5253634218"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "59a8e0ef8560dc9666eaea35e08504d6"
server_key_iv: "415c71c0d9dc9adbb87af02c903cefe5"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.589319 20112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:23.606796 20112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:23.607141 20112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:23.607282 20112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:23.607522 20112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:23.607885 20112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:23.607928 20112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.607959 20112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:23.607980 20112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.614570 20112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44741
I20260924 19:35:23.614639 20239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44741 every 8 connection(s)
I20260924 19:35:23.614936 20112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/data/info.pb
I20260924 19:35:23.616933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 20112
I20260924 19:35:23.617031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-0/wal/instance
I20260924 19:35:23.617237 392 external_mini_cluster.cc:1428] Setting key 7382cac5af4af6bc4cc0c01fcaaf2efc
I20260924 19:35:23.618100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44143
--builtin_ntp_servers=127.0.98.20:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:35:23.620326 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.615619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58589 (local address 127.0.98.62:44143)
0924 19:35:23.615681 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.615683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.615926 (+ 243us) server_negotiation.cc:407] Connection header received
0924 19:35:23.616419 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.616420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.616433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.616489 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:23.617217 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.617882 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.618893 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.619170 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.619235 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.620028 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:23.620030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:23.620030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:23.620034 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:23.620052 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:23.620057 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:23.620108 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.620224 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.620251 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:35:23.620527 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.615605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44143 (local address 127.0.98.1:58589)
0924 19:35:23.615796 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.615830 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.616341 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.616542 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.616546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.616553 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:23.617126 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.617133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.617961 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.617962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.618813 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.618815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.619259 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.619260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.619381 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.619893 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:23.619894 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:23.619922 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:23.619943 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:23.620140 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:23.620147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:23.620150 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.620220 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.620304 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260924 19:35:23.620950 20240 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44143
I20260924 19:35:23.621079 20240 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:23.621279 20240 heartbeater.cc:511] Master 127.0.98.62:44143 requested a full tablet report, sending...
I20260924 19:35:23.621793 20052 ts_manager.cc:194] Registered new tserver with Master: c2007cca0bcb43d29711db5253634218 (127.0.98.1:44741)
I20260924 19:35:23.622402 20052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58589
W20260924 19:35:23.697047 20243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:23.697242 20243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:23.697261 20243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:23.698405 20243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:23.698442 20243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:23.698467 20243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 19:35:23.699812 20243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:23.700030 20243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:23.700305 20243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:23.705039 20248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.705063 20251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.705039 20249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:23.705276 20243 server_base.cc:1034] running on GCE node
I20260924 19:35:23.705472 20243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:23.705703 20243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:23.706848 20243 hybrid_clock.cc:648] HybridClock initialized: now 1790278523706833 us; error 26 us; skew 500 ppm
I20260924 19:35:23.708055 20243 webserver.cc:460] Webserver started at http://127.0.98.2:42483/ using document root <none> and password file <none>
I20260924 19:35:23.708277 20243 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:23.708325 20243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:23.708449 20243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:23.709548 20243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/data/instance:
uuid: "232743f61c6d4842983c831ef2d18fc7"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "c7b9c4bcb37bf7c2429d80a3446bc336"
server_key_iv: "8b4cc9c95e6a885335c6d3868905c2aa"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.709885 20243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/wal/instance:
uuid: "232743f61c6d4842983c831ef2d18fc7"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "c7b9c4bcb37bf7c2429d80a3446bc336"
server_key_iv: "8b4cc9c95e6a885335c6d3868905c2aa"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.711093 20243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:23.711761 20257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.711925 20243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:23.712009 20243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/wal
uuid: "232743f61c6d4842983c831ef2d18fc7"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "c7b9c4bcb37bf7c2429d80a3446bc336"
server_key_iv: "8b4cc9c95e6a885335c6d3868905c2aa"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.712075 20243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:23.730620 20243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:23.730931 20243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:23.731065 20243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:23.731276 20243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:23.731583 20243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:23.731614 20243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.731642 20243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:23.731660 20243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.736931 20243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36295
I20260924 19:35:23.736994 20370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36295 every 8 connection(s)
I20260924 19:35:23.737284 20243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/data/info.pb
I20260924 19:35:23.742321 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.738061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49841 (local address 127.0.98.62:44143)
0924 19:35:23.738117 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.738119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.738299 (+ 180us) server_negotiation.cc:407] Connection header received
0924 19:35:23.738815 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.738816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.738828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.738870 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:23.739538 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.740033 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.741083 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.741275 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.741330 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.742075 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:23.742076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:23.742077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:23.742081 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:23.742097 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:23.742101 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:23.742142 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.742213 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.742240 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:35:23.742514 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.737989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44143 (local address 127.0.98.2:49841)
0924 19:35:23.738177 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.738216 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.738715 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.738947 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.738951 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.738960 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:23.739435 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.739438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.740118 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.740118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.741006 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.741008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.741347 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.741348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.741479 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.741946 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:23.741946 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:23.741976 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:23.741998 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:23.742148 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:23.742153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:23.742156 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.742212 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.742310 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260924 19:35:23.743000 20371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44143
I20260924 19:35:23.743085 20371 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:23.743135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 20243
I20260924 19:35:23.743244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/wal/instance
I20260924 19:35:23.743278 20371 heartbeater.cc:511] Master 127.0.98.62:44143 requested a full tablet report, sending...
I20260924 19:35:23.743428 392 external_mini_cluster.cc:1428] Setting key ed93ee969951dde868b7aa896e41e91c
I20260924 19:35:23.743634 20052 ts_manager.cc:194] Registered new tserver with Master: 232743f61c6d4842983c831ef2d18fc7 (127.0.98.2:36295)
I20260924 19:35:23.744240 20052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49841
I20260924 19:35:23.744544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44143
--builtin_ntp_servers=127.0.98.20:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:35:23.823800 20374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:23.824004 20374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:23.824033 20374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:23.825186 20374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:23.825222 20374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:23.825254 20374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 19:35:23.826560 20374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:23.826790 20374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:23.827057 20374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:23.831789 20382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.831786 20379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:23.831859 20380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:23.832005 20374 server_base.cc:1034] running on GCE node
I20260924 19:35:23.832258 20374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:23.832522 20374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:23.833669 20374 hybrid_clock.cc:648] HybridClock initialized: now 1790278523833672 us; error 46 us; skew 500 ppm
I20260924 19:35:23.834854 20374 webserver.cc:460] Webserver started at http://127.0.98.3:37055/ using document root <none> and password file <none>
I20260924 19:35:23.835079 20374 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:23.835146 20374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:23.835278 20374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:23.836300 20374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/data/instance:
uuid: "686b5a0bb3e4407aa4ad930bf62525fd"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "e23dde15237f9f78290b521e7cab8952"
server_key_iv: "c491fdc2feeba398db0bf8e0232ad39e"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.836614 20374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/wal/instance:
uuid: "686b5a0bb3e4407aa4ad930bf62525fd"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "e23dde15237f9f78290b521e7cab8952"
server_key_iv: "c491fdc2feeba398db0bf8e0232ad39e"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.837814 20374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:23.838513 20388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.838687 20374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:23.838769 20374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/wal
uuid: "686b5a0bb3e4407aa4ad930bf62525fd"
format_stamp: "Formatted at 2026-09-24 19:35:23 on dist-test-slave-m8jb"
server_key: "e23dde15237f9f78290b521e7cab8952"
server_key_iv: "c491fdc2feeba398db0bf8e0232ad39e"
server_key_version: "encryptionkey@0"
I20260924 19:35:23.838821 20374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:23.852910 20374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:23.853176 20374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:23.853278 20374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:23.853466 20374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:35:23.853724 20374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:35:23.853751 20374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.853777 20374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:35:23.853793 20374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:23.858994 20374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41839
I20260924 19:35:23.859037 20501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41839 every 8 connection(s)
I20260924 19:35:23.859328 20374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/data/info.pb
I20260924 19:35:23.864383 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.859962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57007 (local address 127.0.98.62:44143)
0924 19:35:23.860009 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.860011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.860233 (+ 222us) server_negotiation.cc:407] Connection header received
0924 19:35:23.860746 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.860747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.860761 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.860807 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:23.861486 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.861980 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.863063 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.863259 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.863324 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.864097 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:23.864098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:23.864099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:23.864104 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:23.864120 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:23.864125 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:23.864172 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.864254 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.864288 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:35:23.864565 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.859944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44143 (local address 127.0.98.3:57007)
0924 19:35:23.860098 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.860148 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.860624 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.860896 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.860901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.860910 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:23.861390 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.861394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.862128 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.862130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.862966 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.862968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.863336 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.863337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.863470 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.863957 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:35:23.863957 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:23.863984 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:35:23.864007 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:23.864178 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:23.864183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:23.864186 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.864253 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.864354 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260924 19:35:23.865039 20502 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44143
I20260924 19:35:23.865123 20502 heartbeater.cc:464] Registering TS with master...
I20260924 19:35:23.865295 20502 heartbeater.cc:511] Master 127.0.98.62:44143 requested a full tablet report, sending...
I20260924 19:35:23.865710 20052 ts_manager.cc:194] Registered new tserver with Master: 686b5a0bb3e4407aa4ad930bf62525fd (127.0.98.3:41839)
I20260924 19:35:23.866058 20052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57007
I20260924 19:35:23.869048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 20374
I20260924 19:35:23.869135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/wal/instance
I20260924 19:35:23.869304 392 external_mini_cluster.cc:1428] Setting key c817f43f0955b552032178345681a378
I20260924 19:35:23.870303 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:35:23.875008 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.871759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.0.98.62:44143)
0924 19:35:23.871823 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.871826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.872283 (+ 457us) server_negotiation.cc:407] Connection header received
0924 19:35:23.872341 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.872342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.872353 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.872399 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:23.872549 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.873086 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.873831 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.874024 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.874076 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.874759 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:23.874760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:23.874760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:23.874764 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:23.874778 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:23.874782 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:23.874828 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.874901 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.874922 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 19:35:23.876470 20052 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:35:23.877993 20052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:35:23.883967 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.880206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.0.98.2:36295)
0924 19:35:23.880482 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.880486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.880495 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:23.880527 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.880529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.880540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.880602 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:23.880774 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.881635 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.883064 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.883591 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.883651 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.883747 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.883836 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.883876 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":76,"threads_started":1}
I20260924 19:35:23.883953 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.880106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36295 (local address 127.0.0.1:60390)
0924 19:35:23.880261 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.880302 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.880365 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.880623 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.880626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.880648 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:23.880710 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.880712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.881702 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.881702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.882987 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.882989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.883664 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.883665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.883741 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.883754 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.883822 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.883876 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":59,"threads_started":1}
I20260924 19:35:23.884212 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.880117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44741 (local address 127.0.0.1:55168)
0924 19:35:23.880391 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.880402 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.880456 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.880744 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.880745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.880746 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:23.880790 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.880792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.881987 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.881988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.883026 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.883028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.883989 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.883990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.884051 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.884055 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.884099 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.884140 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":105,"threads_started":1}
I20260924 19:35:23.884336 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.880318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41839 (local address 127.0.0.1:58730)
0924 19:35:23.880691 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:23.880700 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:23.880739 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:23.881108 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:23.881109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:23.881111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:23.881147 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.881148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.882276 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.882277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.883117 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:23.883119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.884039 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.884040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:23.884090 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.884092 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 19:35:23.884131 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 19:35:23.884279 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":51,"threads_started":1}
I20260924 19:35:23.884493 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.880228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.0.98.1:44741)
0924 19:35:23.880402 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.880407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.880508 (+ 101us) server_negotiation.cc:407] Connection header received
0924 19:35:23.880566 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.880569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.880583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.880674 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:23.880857 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.881902 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.883105 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.883799 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.883870 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.883954 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.884364 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.884410 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260924 19:35:23.884830 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:23.880496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.0.98.3:41839)
0924 19:35:23.880906 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:23.880909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:23.880919 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:35:23.880962 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:23.880964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:23.880977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:23.881048 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:23.881234 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.882208 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.883190 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:23.883876 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:23.883931 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:23.884013 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:23.884704 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 19:35:23.884750 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":86,"threads_started":1}
I20260924 19:35:23.884922 20305 tablet_service.cc:1467] Processing CreateTablet for tablet c3ee6e877b4c4ca0a704d959fc6b65d6 (DEFAULT_TABLE table=test-table [id=5f1199cd95b44f768df8d5db84c9fe71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:23.885144 20174 tablet_service.cc:1467] Processing CreateTablet for tablet c3ee6e877b4c4ca0a704d959fc6b65d6 (DEFAULT_TABLE table=test-table [id=5f1199cd95b44f768df8d5db84c9fe71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:23.885280 20305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ee6e877b4c4ca0a704d959fc6b65d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:23.885468 20174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ee6e877b4c4ca0a704d959fc6b65d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:23.885440 20436 tablet_service.cc:1467] Processing CreateTablet for tablet c3ee6e877b4c4ca0a704d959fc6b65d6 (DEFAULT_TABLE table=test-table [id=5f1199cd95b44f768df8d5db84c9fe71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:35:23.885731 20436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ee6e877b4c4ca0a704d959fc6b65d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:23.887959 20516 tablet_bootstrap.cc:492] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218: Bootstrap starting.
I20260924 19:35:23.888677 20517 tablet_bootstrap.cc:492] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Bootstrap starting.
I20260924 19:35:23.888803 20518 tablet_bootstrap.cc:492] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: Bootstrap starting.
I20260924 19:35:23.888942 20516 tablet_bootstrap.cc:654] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:23.889287 20516 log.cc:824] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:23.889338 20517 tablet_bootstrap.cc:654] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:23.889392 20518 tablet_bootstrap.cc:654] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:23.889622 20517 log.cc:824] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:23.889690 20518 log.cc:824] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:23.890117 20516 tablet_bootstrap.cc:492] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218: No bootstrap required, opened a new log
I20260924 19:35:23.890187 20516 ts_tablet_manager.cc:1397] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:23.890300 20517 tablet_bootstrap.cc:492] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: No bootstrap required, opened a new log
I20260924 19:35:23.890323 20518 tablet_bootstrap.cc:492] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: No bootstrap required, opened a new log
I20260924 19:35:23.890369 20518 ts_tablet_manager.cc:1397] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:35:23.890369 20517 ts_tablet_manager.cc:1397] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:35:23.892174 20516 raft_consensus.cc:348] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:23.892311 20516 raft_consensus.cc:374] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:23.892342 20516 raft_consensus.cc:729] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2007cca0bcb43d29711db5253634218, State: Initialized, Role: FOLLOWER
I20260924 19:35:23.892473 20516 consensus_queue.cc:260] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:23.892791 20516 ts_tablet_manager.cc:1428] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:35:23.892800 20240 heartbeater.cc:503] Master 127.0.98.62:44143 was elected leader, sending a full tablet report...
I20260924 19:35:23.892997 20517 raft_consensus.cc:348] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:23.893044 20518 raft_consensus.cc:348] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:23.893162 20518 raft_consensus.cc:374] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:23.893162 20517 raft_consensus.cc:374] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:23.893200 20517 raft_consensus.cc:729] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232743f61c6d4842983c831ef2d18fc7, State: Initialized, Role: FOLLOWER
I20260924 19:35:23.893200 20518 raft_consensus.cc:729] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686b5a0bb3e4407aa4ad930bf62525fd, State: Initialized, Role: FOLLOWER
I20260924 19:35:23.893302 20517 consensus_queue.cc:260] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:23.893333 20518 consensus_queue.cc:260] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:23.893564 20518 ts_tablet_manager.cc:1428] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:35:23.893564 20517 ts_tablet_manager.cc:1428] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 19:35:23.893656 20502 heartbeater.cc:503] Master 127.0.98.62:44143 was elected leader, sending a full tablet report...
I20260924 19:35:23.893672 20371 heartbeater.cc:503] Master 127.0.98.62:44143 was elected leader, sending a full tablet report...
W20260924 19:35:23.987812 20372 tablet.cc:2369] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:35:24.009243 20523 raft_consensus.cc:482] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:35:24.009385 20523 raft_consensus.cc:504] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:24.009737 20523 leader_election.cc:288] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2007cca0bcb43d29711db5253634218 (127.0.98.1:44741), 686b5a0bb3e4407aa4ad930bf62525fd (127.0.98.3:41839)
I20260924 19:35:24.012490 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:24.009921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44741 (local address 127.0.98.2:58167)
0924 19:35:24.009971 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:24.009986 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:24.010053 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:24.010212 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:24.010213 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:24.010215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:24.010286 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:24.010288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.010950 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.010951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:24.011733 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:24.011736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.012262 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.012263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:24.012329 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:24.012333 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:35:24.012380 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:35:24.012432 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:35:24.012496 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:24.010042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58167 (local address 127.0.98.1:44741)
0924 19:35:24.010090 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:24.010093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:24.010100 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:24.010128 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:24.010129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:24.010140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:24.010196 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:24.010361 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.010879 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.011813 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.012197 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.012246 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:24.012254 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:24.012390 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 19:35:24.012426 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:35:24.012615 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:24.010109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40991 (local address 127.0.98.3:41839)
0924 19:35:24.010151 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:24.010153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:24.010248 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:35:24.010298 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:24.010299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:24.010308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:24.010356 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:35:24.010487 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.011025 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.011918 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.012283 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.012345 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:24.012354 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:24.012516 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 19:35:24.012556 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:35:24.012773 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:24.010017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41839 (local address 127.0.98.2:40991)
0924 19:35:24.010189 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:24.010197 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:24.010243 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:24.010367 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:24.010367 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:24.010369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:35:24.010430 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:24.010431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.011100 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.011101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:24.011832 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:24.011834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.012356 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.012357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:24.012440 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:24.012443 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 19:35:24.012504 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:35:24.012718 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260924 19:35:24.012837 20194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ee6e877b4c4ca0a704d959fc6b65d6" candidate_uuid: "232743f61c6d4842983c831ef2d18fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2007cca0bcb43d29711db5253634218" is_pre_election: true
I20260924 19:35:24.012987 20194 raft_consensus.cc:2454] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 232743f61c6d4842983c831ef2d18fc7 in term 0.
I20260924 19:35:24.013216 20260 leader_election.cc:302] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232743f61c6d4842983c831ef2d18fc7, c2007cca0bcb43d29711db5253634218; no voters:
I20260924 19:35:24.013341 20456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ee6e877b4c4ca0a704d959fc6b65d6" candidate_uuid: "232743f61c6d4842983c831ef2d18fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" is_pre_election: true
I20260924 19:35:24.013365 20523 raft_consensus.cc:2781] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:35:24.013417 20523 raft_consensus.cc:482] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:35:24.013434 20523 raft_consensus.cc:3037] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:24.013470 20456 raft_consensus.cc:2454] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 232743f61c6d4842983c831ef2d18fc7 in term 0.
I20260924 19:35:24.014145 20523 raft_consensus.cc:504] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:24.014251 20523 leader_election.cc:288] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [CANDIDATE]: Term 1 election: Requested vote from peers c2007cca0bcb43d29711db5253634218 (127.0.98.1:44741), 686b5a0bb3e4407aa4ad930bf62525fd (127.0.98.3:41839)
I20260924 19:35:24.014472 20194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ee6e877b4c4ca0a704d959fc6b65d6" candidate_uuid: "232743f61c6d4842983c831ef2d18fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2007cca0bcb43d29711db5253634218"
I20260924 19:35:24.014519 20456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ee6e877b4c4ca0a704d959fc6b65d6" candidate_uuid: "232743f61c6d4842983c831ef2d18fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686b5a0bb3e4407aa4ad930bf62525fd"
I20260924 19:35:24.014554 20194 raft_consensus.cc:3037] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:24.014578 20456 raft_consensus.cc:3037] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:24.015479 20194 raft_consensus.cc:2454] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 232743f61c6d4842983c831ef2d18fc7 in term 1.
I20260924 19:35:24.015479 20456 raft_consensus.cc:2454] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 232743f61c6d4842983c831ef2d18fc7 in term 1.
I20260924 19:35:24.015636 20260 leader_election.cc:302] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232743f61c6d4842983c831ef2d18fc7, 686b5a0bb3e4407aa4ad930bf62525fd; no voters:
I20260924 19:35:24.015758 20523 raft_consensus.cc:2781] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:24.015887 20523 raft_consensus.cc:686] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 1 LEADER]: Becoming Leader. State: Replica: 232743f61c6d4842983c831ef2d18fc7, State: Running, Role: LEADER
I20260924 19:35:24.015967 20523 consensus_queue.cc:237] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } }
I20260924 19:35:24.016801 20051 catalog_manager.cc:5495] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 232743f61c6d4842983c831ef2d18fc7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "232743f61c6d4842983c831ef2d18fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232743f61c6d4842983c831ef2d18fc7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 } health_report { overall_health: UNKNOWN } } }
I20260924 19:35:24.031553 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:24.029441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.98.2:36295)
0924 19:35:24.029480 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:24.029482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:24.029492 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:35:24.029527 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:24.029528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:24.029543 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:24.029608 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:24.029692 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.030166 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.030842 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:24.031044 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:24.031087 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:24.031243 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:24.031362 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:24.031424 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:24.031454 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 19:35:24.031478 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:24.033180 20456 raft_consensus.cc:1261] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 1 FOLLOWER]: Refusing update from remote peer 232743f61c6d4842983c831ef2d18fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:24.033200 20194 raft_consensus.cc:1261] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 1 FOLLOWER]: Refusing update from remote peer 232743f61c6d4842983c831ef2d18fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:24.033490 20523 consensus_queue.cc:1035] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2007cca0bcb43d29711db5253634218" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:24.033615 20526 consensus_queue.cc:1035] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "686b5a0bb3e4407aa4ad930bf62525fd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:24.036360 20531 mvcc.cc:205] Tried to move back new op lower bound from 7332980834437545984 to 7332980834370220032. Current Snapshot: MvccSnapshot[applied={T|T < 7332980834437545984}]
I20260924 19:35:24.037886 20051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:24.037967 20051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:35:24.038631 20051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f327b0c1be44917a8dfe74465495834: Sending DeleteTablet for 3 replicas of tablet c3ee6e877b4c4ca0a704d959fc6b65d6
I20260924 19:35:24.038945 20305 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ee6e877b4c4ca0a704d959fc6b65d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:24 UTC) from {username='slave'} at 127.0.0.1:60390
I20260924 19:35:24.038955 20436 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ee6e877b4c4ca0a704d959fc6b65d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:24 UTC) from {username='slave'} at 127.0.0.1:58730
I20260924 19:35:24.039152 20539 tablet_replica.cc:330] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: stopping tablet replica
I20260924 19:35:24.039150 20174 tablet_service.cc:1514] Processing DeleteTablet for tablet c3ee6e877b4c4ca0a704d959fc6b65d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:35:24 UTC) from {username='slave'} at 127.0.0.1:55168
I20260924 19:35:24.039247 20539 raft_consensus.cc:2229] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:35:24.039371 20539 raft_consensus.cc:2258] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:24.039377 20541 tablet_replica.cc:330] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218: stopping tablet replica
I20260924 19:35:24.039556 20541 raft_consensus.cc:2229] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:24.039700 20541 raft_consensus.cc:2258] T c3ee6e877b4c4ca0a704d959fc6b65d6 P c2007cca0bcb43d29711db5253634218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:24.039770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 20112
I20260924 19:35:24.040313 20539 ts_tablet_manager.cc:1905] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:24.041458 20540 tablet_replica.cc:330] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: stopping tablet replica
I20260924 19:35:24.041549 20540 raft_consensus.cc:2229] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:35:24.041642 20540 raft_consensus.cc:2258] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:35:24.041994 20539 ts_tablet_manager.cc:1918] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:35:24.042089 20540 ts_tablet_manager.cc:1905] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:35:24.042109 20539 log.cc:1196] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-1/wal/wals/c3ee6e877b4c4ca0a704d959fc6b65d6
I20260924 19:35:24.042382 20539 ts_tablet_manager.cc:1939] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 232743f61c6d4842983c831ef2d18fc7: Deleting consensus metadata
I20260924 19:35:24.042758 20036 catalog_manager.cc:4841] TS 232743f61c6d4842983c831ef2d18fc7 (127.0.98.2:36295): tablet c3ee6e877b4c4ca0a704d959fc6b65d6 (table test-table [id=5f1199cd95b44f768df8d5db84c9fe71]) successfully deleted
I20260924 19:35:24.043481 20540 ts_tablet_manager.cc:1918] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:35:24.043538 20540 log.cc:1196] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: Deleting WAL directory at /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790278290121024-392-0/minicluster-data/ts-2/wal/wals/c3ee6e877b4c4ca0a704d959fc6b65d6
I20260924 19:35:24.043722 20540 ts_tablet_manager.cc:1939] T c3ee6e877b4c4ca0a704d959fc6b65d6 P 686b5a0bb3e4407aa4ad930bf62525fd: Deleting consensus metadata
I20260924 19:35:24.044044 20038 catalog_manager.cc:4841] TS 686b5a0bb3e4407aa4ad930bf62525fd (127.0.98.3:41839): tablet c3ee6e877b4c4ca0a704d959fc6b65d6 (table test-table [id=5f1199cd95b44f768df8d5db84c9fe71]) successfully deleted
W20260924 19:35:24.046105 20038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44741 (error 108)
I20260924 19:35:24.046286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 20243
I20260924 19:35:24.047117 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:24.046463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44741 (local address 127.0.0.1:55184)
0924 19:35:24.047035 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:24.047083 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549}
W20260924 19:35:24.047269 20038 catalog_manager.cc:4584] TS c2007cca0bcb43d29711db5253634218 (127.0.98.1:44741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3ee6e877b4c4ca0a704d959fc6b65d6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44741: connect: Connection refused (error 111)
I20260924 19:35:24.051084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 20374
I20260924 19:35:24.055193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 20021
2026-09-24T19:35:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (729 ms)
[----------] 6 tests from EncryptionPolicyTest (8120 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:24 dist-test-slave-m8jb krb5kdc[20545](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:24 dist-test-slave-m8jb krb5kdc[20545](info): set up 2 sockets
Sep 24 19:35:24 dist-test-slave-m8jb krb5kdc[20545](info): commencing operation
krb5kdc: starting...
W20260924 19:35:25.004339 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:25 dist-test-slave-m8jb krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:25.116484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:25.196514 20558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:25.196769 20558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:25.196795 20558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:25.198002 20558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:25.198051 20558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:25.198067 20558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:25.198081 20558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:25.198096 20558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:25.199618 20558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:25.199851 20558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:25.200080 20558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:25.204869 20563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:25.204880 20558 server_base.cc:1034] running on GCE node
W20260924 19:35:25.204854 20564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:25.204852 20566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:25.205248 20558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:25.205488 20558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:25.206637 20558 hybrid_clock.cc:648] HybridClock initialized: now 1790278525206623 us; error 27 us; skew 500 ppm
Sep 24 19:35:25 dist-test-slave-m8jb krb5kdc[20545](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:25.209050 20558 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:25.209463 20558 webserver.cc:460] Webserver started at http://127.0.0.1:40885/ using document root <none> and password file <none>
I20260924 19:35:25.209651 20558 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:25.209693 20558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:25.209847 20558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:25.210868 20558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "531808098c184a45a131fe2357d13ad0"
format_stamp: "Formatted at 2026-09-24 19:35:25 on dist-test-slave-m8jb"
server_key: "78a069d4628a7bb2ee3dbcb9ad0286ea"
server_key_iv: "731fd6658b9351056c0b79bc4ffeef26"
server_key_version: "encryptionkey@0"
I20260924 19:35:25.211189 20558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "531808098c184a45a131fe2357d13ad0"
format_stamp: "Formatted at 2026-09-24 19:35:25 on dist-test-slave-m8jb"
server_key: "78a069d4628a7bb2ee3dbcb9ad0286ea"
server_key_iv: "731fd6658b9351056c0b79bc4ffeef26"
server_key_version: "encryptionkey@0"
I20260924 19:35:25.212419 20558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:25.213212 20573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:25.213352 20558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:25.213438 20558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "531808098c184a45a131fe2357d13ad0"
format_stamp: "Formatted at 2026-09-24 19:35:25 on dist-test-slave-m8jb"
server_key: "78a069d4628a7bb2ee3dbcb9ad0286ea"
server_key_iv: "731fd6658b9351056c0b79bc4ffeef26"
server_key_version: "encryptionkey@0"
I20260924 19:35:25.213500 20558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:25.251437 20558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:25.253559 20558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:25.253696 20558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:25.257300 20558 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42033
I20260924 19:35:25.257645 20558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:25.257632 20625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42033 every 8 connection(s)
I20260924 19:35:25.258304 20626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:25.260694 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0: Bootstrap starting.
I20260924 19:35:25.261260 20626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:25.261451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 20558
I20260924 19:35:25.261572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:25.261608 20626 log.cc:824] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:25.261808 392 external_mini_cluster.cc:1428] Setting key 528a43fe48a05198c41796938728acc0
I20260924 19:35:25.262459 20626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0: No bootstrap required, opened a new log
I20260924 19:35:25.264704 20626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531808098c184a45a131fe2357d13ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260924 19:35:25.264850 20626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:25.264881 20626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531808098c184a45a131fe2357d13ad0, State: Initialized, Role: FOLLOWER
I20260924 19:35:25.265002 20626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531808098c184a45a131fe2357d13ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260924 19:35:25.265074 20626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:25.265113 20626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:25.265161 20626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:25.265717 20626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531808098c184a45a131fe2357d13ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260924 19:35:25.265807 20626 leader_election.cc:302] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531808098c184a45a131fe2357d13ad0; no voters:
I20260924 19:35:25.265974 20626 leader_election.cc:288] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:25.266012 20631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:25.266124 20631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [term 1 LEADER]: Becoming Leader. State: Replica: 531808098c184a45a131fe2357d13ad0, State: Running, Role: LEADER
I20260924 19:35:25.266201 20631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531808098c184a45a131fe2357d13ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } }
I20260924 19:35:25.266232 20626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:25.266825 20633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531808098c184a45a131fe2357d13ad0. Latest consensus state: current_term: 1 leader_uuid: "531808098c184a45a131fe2357d13ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531808098c184a45a131fe2357d13ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } } }
I20260924 19:35:25.266837 20632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531808098c184a45a131fe2357d13ad0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531808098c184a45a131fe2357d13ad0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42033 } } }
I20260924 19:35:25.267252 20632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:25.267462 20633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:25.267448 20643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:25.268242 20643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:35:25 dist-test-slave-m8jb krb5kdc[20545](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:35:25.269831 20643 catalog_manager.cc:1347] Generated new cluster ID: 5976582118454239b14dd02daf2b646e
I20260924 19:35:25.269932 20643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:25.273126 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:25.262355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59601 (local address 127.0.0.1:42033)
0924 19:35:25.262523 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:25.262532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:25.262565 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:35:25.263023 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:25.263029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:25.263074 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:25.263200 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:25.263821 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:25.264732 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:25.265622 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:25.266138 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:25.266546 (+ 408us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:25.268960 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:25.268964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:25.268965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:25.268976 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:25.270927 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:25.271411 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:25.271415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:25.271415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:25.271524 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:25.271780 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:25.271781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:25.271781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:25.272112 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:25.272274 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:25.272780 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 19:35:25.272907 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260924 19:35:25.274050 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:25.282534 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:25.275616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41971 (local address 127.0.0.1:42033)
0924 19:35:25.275652 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:25.275653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:25.275709 (+ 56us) server_negotiation.cc:407] Connection header received
0924 19:35:25.275776 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:25.275777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:25.275786 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:25.275842 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:25.276407 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:25.277222 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:25.278183 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:25.278507 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:25.278592 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:25.279892 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:25.279893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:25.279894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:25.279901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:25.281417 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:25.281792 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:25.281793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:25.281794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:25.281851 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:25.282041 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:25.282041 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:25.282042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:25.282180 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:25.282277 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:25.282376 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:35:25.282438 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:25.287772 20643 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:25.288267 20643 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:25.294262 20643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531808098c184a45a131fe2357d13ad0: Generated new TSK 0
I20260924 19:35:25.294433 20643 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:25.301388 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 20558
W20260924 19:35:25.305215 20543 connection.cc:535] client connection to 127.0.0.1:42033 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42033): Connection reset by peer (error 104)
2026-09-24T19:35:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:25 dist-test-slave-m8jb krb5kdc[20657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:25 dist-test-slave-m8jb krb5kdc[20657](info): set up 2 sockets
Sep 24 19:35:25 dist-test-slave-m8jb krb5kdc[20657](info): commencing operation
krb5kdc: starting...
W20260924 19:35:26.248427 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:26 dist-test-slave-m8jb krb5kdc[20657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:26.361238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:26.440177 20670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:26.440385 20670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:26.440408 20670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:26.441577 20670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:26.441622 20670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:26.441634 20670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:26.441645 20670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:26.441656 20670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:26.443156 20670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45063
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:26.443391 20670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:26.443601 20670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:26.448274 20678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:26.448318 20675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:26.448282 20676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:26.448486 20670 server_base.cc:1034] running on GCE node
I20260924 19:35:26.448750 20670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:26.449082 20670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:26.450224 20670 hybrid_clock.cc:648] HybridClock initialized: now 1790278526450205 us; error 35 us; skew 500 ppm
Sep 24 19:35:26 dist-test-slave-m8jb krb5kdc[20657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:26.452664 20670 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:26.453155 20670 webserver.cc:460] Webserver started at http://127.0.0.1:37027/ using document root <none> and password file <none>
I20260924 19:35:26.453366 20670 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:26.453413 20670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:26.453537 20670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:26.454555 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "ddef6068af274d2b8af4cc58f95dad22"
format_stamp: "Formatted at 2026-09-24 19:35:26 on dist-test-slave-m8jb"
server_key: "9faa3fa39b871f7091001e0a8d07515d"
server_key_iv: "0911bdcf4646f98543ca25924baf3cd8"
server_key_version: "encryptionkey@0"
I20260924 19:35:26.454893 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "ddef6068af274d2b8af4cc58f95dad22"
format_stamp: "Formatted at 2026-09-24 19:35:26 on dist-test-slave-m8jb"
server_key: "9faa3fa39b871f7091001e0a8d07515d"
server_key_iv: "0911bdcf4646f98543ca25924baf3cd8"
server_key_version: "encryptionkey@0"
I20260924 19:35:26.456188 20670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:35:26.456940 20685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:26.457082 20670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:26.457166 20670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "ddef6068af274d2b8af4cc58f95dad22"
format_stamp: "Formatted at 2026-09-24 19:35:26 on dist-test-slave-m8jb"
server_key: "9faa3fa39b871f7091001e0a8d07515d"
server_key_iv: "0911bdcf4646f98543ca25924baf3cd8"
server_key_version: "encryptionkey@0"
I20260924 19:35:26.457229 20670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:26.474673 20670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:26.476534 20670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:26.476689 20670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:26.480376 20670 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45063
I20260924 19:35:26.480427 20737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45063 every 8 connection(s)
I20260924 19:35:26.480801 20670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:26.481617 20738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:26.483920 20738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22: Bootstrap starting.
I20260924 19:35:26.484470 20738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:26.484742 20738 log.cc:824] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:26.485432 20738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22: No bootstrap required, opened a new log
I20260924 19:35:26.485740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 20670
I20260924 19:35:26.485847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:26.486016 392 external_mini_cluster.cc:1428] Setting key b5801589b1ad355abb2a3420a72d7b77
I20260924 19:35:26.487546 20738 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddef6068af274d2b8af4cc58f95dad22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45063 } }
I20260924 19:35:26.487686 20738 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:26.487717 20738 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddef6068af274d2b8af4cc58f95dad22, State: Initialized, Role: FOLLOWER
I20260924 19:35:26.487895 20738 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddef6068af274d2b8af4cc58f95dad22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45063 } }
I20260924 19:35:26.487972 20738 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:26.488010 20738 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:26.488055 20738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:26.488595 20738 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddef6068af274d2b8af4cc58f95dad22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45063 } }
I20260924 19:35:26.488718 20738 leader_election.cc:302] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddef6068af274d2b8af4cc58f95dad22; no voters:
I20260924 19:35:26.488875 20738 leader_election.cc:288] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:26.488915 20743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:26.489069 20743 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [term 1 LEADER]: Becoming Leader. State: Replica: ddef6068af274d2b8af4cc58f95dad22, State: Running, Role: LEADER
I20260924 19:35:26.489120 20738 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:26.489154 20743 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddef6068af274d2b8af4cc58f95dad22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45063 } }
I20260924 19:35:26.489483 20743 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddef6068af274d2b8af4cc58f95dad22. Latest consensus state: current_term: 1 leader_uuid: "ddef6068af274d2b8af4cc58f95dad22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddef6068af274d2b8af4cc58f95dad22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45063 } } }
I20260924 19:35:26.489537 20744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddef6068af274d2b8af4cc58f95dad22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddef6068af274d2b8af4cc58f95dad22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45063 } } }
I20260924 19:35:26.489617 20744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:26.489603 20743 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:26.489964 20752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:26.490654 20752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:26.492281 20752 catalog_manager.cc:1347] Generated new cluster ID: be4bbcc9611241d7ae647cfdd2822d77
I20260924 19:35:26.492331 20752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:26.492353 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:26.486492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42875 (local address 127.0.0.1:45063)
0924 19:35:26.486649 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:26.486656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:26.486689 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:35:26.487117 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:26.487123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:26.487168 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:26.487275 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:26.487524 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:26.488862 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:26.489965 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:26.490547 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:26.490645 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:26.491689 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:26.491692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:26.491693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:26.491894 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:26.491932 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:26.491946 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:26.492023 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:26.492127 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 19:35:26.492154 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260924 19:35:26.493319 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:26.498098 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:26.494693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39969 (local address 127.0.0.1:45063)
0924 19:35:26.494734 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:26.494735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:26.494986 (+ 251us) server_negotiation.cc:407] Connection header received
0924 19:35:26.495028 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:26.495029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:26.495038 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:26.495100 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:26.495209 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:26.495919 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:26.496885 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:26.497188 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:26.497255 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:26.497875 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:26.497877 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:26.497878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:26.497883 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:26.497898 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:26.497902 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:26.497956 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:26.498017 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:35:26.498027 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:26.513764 20752 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:26.514276 20752 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:26.518927 20752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddef6068af274d2b8af4cc58f95dad22: Generated new TSK 0
I20260924 19:35:26.519085 20752 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:26.523716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 20670
W20260924 19:35:26.527417 20655 connection.cc:535] client connection to 127.0.0.1:45063 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45063): Connection reset by peer (error 104)
2026-09-24T19:35:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:26 dist-test-slave-m8jb krb5kdc[20769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:26 dist-test-slave-m8jb krb5kdc[20769](info): set up 2 sockets
Sep 24 19:35:26 dist-test-slave-m8jb krb5kdc[20769](info): commencing operation
krb5kdc: starting...
W20260924 19:35:27.468454 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:27 dist-test-slave-m8jb krb5kdc[20769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:27.583205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33303
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:27.663621 20782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:27.663834 20782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:27.663856 20782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:27.665025 20782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:27.665067 20782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:27.665081 20782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:27.665094 20782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:27.665117 20782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:27.666623 20782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:27.666859 20782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:27.667066 20782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:27.671837 20790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:27.671924 20782 server_base.cc:1034] running on GCE node
W20260924 19:35:27.671815 20787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:27.671815 20788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:27.672292 20782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:27.672565 20782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:27.673709 20782 hybrid_clock.cc:648] HybridClock initialized: now 1790278527673706 us; error 40 us; skew 500 ppm
Sep 24 19:35:27 dist-test-slave-m8jb krb5kdc[20769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:27.676186 20782 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:27.676601 20782 webserver.cc:460] Webserver started at http://127.0.0.1:34569/ using document root <none> and password file <none>
I20260924 19:35:27.676841 20782 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:27.676889 20782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:27.677013 20782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:27.678117 20782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "34ec54a8a323434ba390caefaa2db76d"
format_stamp: "Formatted at 2026-09-24 19:35:27 on dist-test-slave-m8jb"
server_key: "b8a9f7c9e4f86fc6c2e02c7c696046de"
server_key_iv: "7a5418a6c3f84d5571b582eef990d228"
server_key_version: "encryptionkey@0"
I20260924 19:35:27.678454 20782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "34ec54a8a323434ba390caefaa2db76d"
format_stamp: "Formatted at 2026-09-24 19:35:27 on dist-test-slave-m8jb"
server_key: "b8a9f7c9e4f86fc6c2e02c7c696046de"
server_key_iv: "7a5418a6c3f84d5571b582eef990d228"
server_key_version: "encryptionkey@0"
I20260924 19:35:27.679701 20782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:35:27.680444 20797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:27.680653 20782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:35:27.680755 20782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "34ec54a8a323434ba390caefaa2db76d"
format_stamp: "Formatted at 2026-09-24 19:35:27 on dist-test-slave-m8jb"
server_key: "b8a9f7c9e4f86fc6c2e02c7c696046de"
server_key_iv: "7a5418a6c3f84d5571b582eef990d228"
server_key_version: "encryptionkey@0"
I20260924 19:35:27.680826 20782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:27.683928 20782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:27.685760 20782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:27.685880 20782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:27.689411 20782 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33303
I20260924 19:35:27.689442 20849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33303 every 8 connection(s)
I20260924 19:35:27.689726 20782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:27.690739 20850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:27.693126 20850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d: Bootstrap starting.
I20260924 19:35:27.693794 20850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:27.694072 20850 log.cc:824] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:27.694761 20850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d: No bootstrap required, opened a new log
I20260924 19:35:27.696408 20850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ec54a8a323434ba390caefaa2db76d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33303 } }
I20260924 19:35:27.696550 20850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:27.696609 20850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ec54a8a323434ba390caefaa2db76d, State: Initialized, Role: FOLLOWER
I20260924 19:35:27.696786 20850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ec54a8a323434ba390caefaa2db76d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33303 } }
I20260924 19:35:27.696863 20850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:27.696898 20850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:27.696947 20850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:27.697480 20850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ec54a8a323434ba390caefaa2db76d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33303 } }
I20260924 19:35:27.697574 20850 leader_election.cc:302] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ec54a8a323434ba390caefaa2db76d; no voters:
I20260924 19:35:27.697679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 20782
I20260924 19:35:27.697777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:27.697777 20850 leader_election.cc:288] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:27.697925 20854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:27.697956 392 external_mini_cluster.cc:1428] Setting key 9283dde3ced245ece8ca0656434a6cf4
I20260924 19:35:27.698020 20850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:27.698261 20854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [term 1 LEADER]: Becoming Leader. State: Replica: 34ec54a8a323434ba390caefaa2db76d, State: Running, Role: LEADER
I20260924 19:35:27.698591 20854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ec54a8a323434ba390caefaa2db76d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33303 } }
I20260924 19:35:27.699039 20853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34ec54a8a323434ba390caefaa2db76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ec54a8a323434ba390caefaa2db76d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33303 } } }
I20260924 19:35:27.699136 20853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:27.699347 20855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ec54a8a323434ba390caefaa2db76d. Latest consensus state: current_term: 1 leader_uuid: "34ec54a8a323434ba390caefaa2db76d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ec54a8a323434ba390caefaa2db76d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33303 } } }
I20260924 19:35:27.699415 20855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d [sys.catalog]: This master's current role is: LEADER
W20260924 19:35:27.699909 20870 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:27.699955 20870 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:27.700032 20867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:27.700212 20867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:27.701686 20867 catalog_manager.cc:1347] Generated new cluster ID: 8815a5fe7b2c4ba1bd3ad5f7c16cad63
I20260924 19:35:27.701745 20867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:27.706107 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:27.698419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48189 (local address 127.0.0.1:33303)
0924 19:35:27.698583 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:27.698593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:27.698628 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:35:27.698745 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:27.698753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:27.698782 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:27.699169 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:27.705664 (+ 6495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:27.705668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:27.705668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:27.705708 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:27.705747 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:27.705759 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:27.705813 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:27.705881 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 19:35:27.705921 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260924 19:35:27.706925 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:27.709208 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:27.708204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33337 (local address 127.0.0.1:33303)
0924 19:35:27.708241 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:27.708242 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:27.708364 (+ 122us) server_negotiation.cc:407] Connection header received
0924 19:35:27.708413 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:27.708414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:27.708438 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:27.708481 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:27.709004 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:27.709005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:27.709005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:27.709009 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:27.709023 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:27.709027 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:27.709071 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:27.709119 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 19:35:27.709133 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:35:27.726512 20867 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:27.726774 20867 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:27.734591 20867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34ec54a8a323434ba390caefaa2db76d: Generated new TSK 0
I20260924 19:35:27.734761 20867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:27.743165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 20782
2026-09-24T19:35:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:27 dist-test-slave-m8jb krb5kdc[20881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:27 dist-test-slave-m8jb krb5kdc[20881](info): set up 2 sockets
Sep 24 19:35:27 dist-test-slave-m8jb krb5kdc[20881](info): commencing operation
krb5kdc: starting...
W20260924 19:35:28.688490 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:28 dist-test-slave-m8jb krb5kdc[20881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:28.799464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:28.878005 20894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:28.878216 20894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:28.878239 20894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:28.879421 20894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:28.879462 20894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:28.879475 20894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:28.879485 20894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:28.879496 20894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:28.881009 20894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:28.881238 20894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:28.881446 20894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:28.886142 20902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:28.886265 20894 server_base.cc:1034] running on GCE node
W20260924 19:35:28.886142 20899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:28.886366 20900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:28.886648 20894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:28.886929 20894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:28.888079 20894 hybrid_clock.cc:648] HybridClock initialized: now 1790278528888063 us; error 30 us; skew 500 ppm
Sep 24 19:35:28 dist-test-slave-m8jb krb5kdc[20881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:28.890591 20894 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:28.891000 20894 webserver.cc:460] Webserver started at http://127.0.0.1:34559/ using document root <none> and password file <none>
I20260924 19:35:28.891193 20894 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:28.891233 20894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:28.891324 20894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:28.892300 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "32bd53de890b41bda271599f3f07ee3d"
format_stamp: "Formatted at 2026-09-24 19:35:28 on dist-test-slave-m8jb"
server_key: "d10d306a57a54f3141c412e811bbabc1"
server_key_iv: "71af278590a2ef349f6a50da9f0f7bf7"
server_key_version: "encryptionkey@0"
I20260924 19:35:28.892592 20894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "32bd53de890b41bda271599f3f07ee3d"
format_stamp: "Formatted at 2026-09-24 19:35:28 on dist-test-slave-m8jb"
server_key: "d10d306a57a54f3141c412e811bbabc1"
server_key_iv: "71af278590a2ef349f6a50da9f0f7bf7"
server_key_version: "encryptionkey@0"
I20260924 19:35:28.893836 20894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:28.894482 20909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:28.894662 20894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:28.894747 20894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "32bd53de890b41bda271599f3f07ee3d"
format_stamp: "Formatted at 2026-09-24 19:35:28 on dist-test-slave-m8jb"
server_key: "d10d306a57a54f3141c412e811bbabc1"
server_key_iv: "71af278590a2ef349f6a50da9f0f7bf7"
server_key_version: "encryptionkey@0"
I20260924 19:35:28.894814 20894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:28.907603 20894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:28.909684 20894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:28.909804 20894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:28.913633 20894 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35815
I20260924 19:35:28.913671 20961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35815 every 8 connection(s)
I20260924 19:35:28.913967 20894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:28.914747 20962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:28.916970 20962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d: Bootstrap starting.
I20260924 19:35:28.917480 20962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:28.917692 20962 log.cc:824] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:28.918375 20962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d: No bootstrap required, opened a new log
I20260924 19:35:28.919906 20962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd53de890b41bda271599f3f07ee3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260924 19:35:28.920071 20962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:28.920097 20962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bd53de890b41bda271599f3f07ee3d, State: Initialized, Role: FOLLOWER
I20260924 19:35:28.920208 20962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd53de890b41bda271599f3f07ee3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260924 19:35:28.920282 20962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:28.920308 20962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:28.920351 20962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:28.920892 20962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd53de890b41bda271599f3f07ee3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260924 19:35:28.920979 20962 leader_election.cc:302] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32bd53de890b41bda271599f3f07ee3d; no voters:
I20260924 19:35:28.921105 20962 leader_election.cc:288] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:28.921173 20965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:28.921336 20962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:28.921347 20965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [term 1 LEADER]: Becoming Leader. State: Replica: 32bd53de890b41bda271599f3f07ee3d, State: Running, Role: LEADER
I20260924 19:35:28.921456 20965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd53de890b41bda271599f3f07ee3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } }
I20260924 19:35:28.921829 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32bd53de890b41bda271599f3f07ee3d. Latest consensus state: current_term: 1 leader_uuid: "32bd53de890b41bda271599f3f07ee3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd53de890b41bda271599f3f07ee3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } } }
I20260924 19:35:28.921849 20966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32bd53de890b41bda271599f3f07ee3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32bd53de890b41bda271599f3f07ee3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35815 } } }
I20260924 19:35:28.921988 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:28.922081 20966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:28.922367 20974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:28.922869 20974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:28.923961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 20894
I20260924 19:35:28.924072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:28.924244 392 external_mini_cluster.cc:1428] Setting key fb271a407d8f651b6bee38c23b9181eb
I20260924 19:35:28.924494 20974 catalog_manager.cc:1347] Generated new cluster ID: acf879e076ed4fe4be5c8c35e3794d86
I20260924 19:35:28.924542 20974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 19:35:28 dist-test-slave-m8jb krb5kdc[20881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:35:28.935370 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:28.924867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59329 (local address 127.0.0.1:35815)
0924 19:35:28.925027 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:28.925036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:28.925073 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:35:28.925230 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:28.925265 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:28.925322 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:28.925473 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:28.926115 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:28.927580 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:28.928574 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:28.929214 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:28.929342 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:28.932027 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:28.932031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:28.932031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:28.932043 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:28.933841 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:28.934213 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:28.934215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:28.934216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:28.934311 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:28.934482 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:28.934482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:28.934483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:28.934770 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:28.934889 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:28.934999 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 19:35:28.935123 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260924 19:35:28.936219 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:28.944623 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:28.937654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56811 (local address 127.0.0.1:35815)
0924 19:35:28.937687 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:28.937689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:28.937798 (+ 109us) server_negotiation.cc:407] Connection header received
0924 19:35:28.937838 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:28.937839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:28.937863 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:28.937920 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:28.938461 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:28.939196 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:28.940128 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:28.940459 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:28.940520 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:28.941813 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:28.941814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:28.941815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:28.941822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:28.943269 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:28.943594 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:28.943595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:28.943595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:28.943648 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:28.943799 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:28.943800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:28.943800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:28.943922 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:28.944417 (+ 495us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:28.944455 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 19:35:28.944530 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:35:28.951104 20974 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:28.951835 20974 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:28.960579 20974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32bd53de890b41bda271599f3f07ee3d: Generated new TSK 0
I20260924 19:35:28.960794 20974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:28.965785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 20894
2026-09-24T19:35:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1222 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:29 dist-test-slave-m8jb krb5kdc[20993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:29 dist-test-slave-m8jb krb5kdc[20993](info): set up 2 sockets
Sep 24 19:35:29 dist-test-slave-m8jb krb5kdc[20993](info): commencing operation
krb5kdc: starting...
W20260924 19:35:29.908378 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:29 dist-test-slave-m8jb krb5kdc[20993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:30.023954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:30.102825 21006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:30.103042 21006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:30.103073 21006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:30.104259 21006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:30.104305 21006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:30.104321 21006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:30.104337 21006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:30.104352 21006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:30.105901 21006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:30.106150 21006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:30.106356 21006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:30.111150 21011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:30.111128 21012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:30.111119 21014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:30.111310 21006 server_base.cc:1034] running on GCE node
I20260924 19:35:30.111653 21006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:30.111953 21006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:30.113116 21006 hybrid_clock.cc:648] HybridClock initialized: now 1790278530113092 us; error 34 us; skew 500 ppm
Sep 24 19:35:30 dist-test-slave-m8jb krb5kdc[20993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:30.115485 21006 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:30.115933 21006 webserver.cc:460] Webserver started at http://127.0.0.1:42529/ using document root <none> and password file <none>
I20260924 19:35:30.116124 21006 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:30.116170 21006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:30.116299 21006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:30.117344 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "261fb9053f9d4272925cf94a06e10c8d"
format_stamp: "Formatted at 2026-09-24 19:35:30 on dist-test-slave-m8jb"
server_key: "635522db323188c7ca1b04909cc64fee"
server_key_iv: "c8737eb0b6616c6c34a1ffb6896faf9c"
server_key_version: "encryptionkey@0"
I20260924 19:35:30.117661 21006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "261fb9053f9d4272925cf94a06e10c8d"
format_stamp: "Formatted at 2026-09-24 19:35:30 on dist-test-slave-m8jb"
server_key: "635522db323188c7ca1b04909cc64fee"
server_key_iv: "c8737eb0b6616c6c34a1ffb6896faf9c"
server_key_version: "encryptionkey@0"
I20260924 19:35:30.118876 21006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:30.119505 21021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:30.119648 21006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:30.119727 21006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "261fb9053f9d4272925cf94a06e10c8d"
format_stamp: "Formatted at 2026-09-24 19:35:30 on dist-test-slave-m8jb"
server_key: "635522db323188c7ca1b04909cc64fee"
server_key_iv: "c8737eb0b6616c6c34a1ffb6896faf9c"
server_key_version: "encryptionkey@0"
I20260924 19:35:30.119786 21006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:30.128285 21006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:30.130127 21006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:30.130264 21006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:30.133903 21006 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40537
I20260924 19:35:30.133948 21073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40537 every 8 connection(s)
I20260924 19:35:30.134261 21006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:30.135097 21074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:30.137326 21074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d: Bootstrap starting.
I20260924 19:35:30.137832 21074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:30.138048 21074 log.cc:824] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:30.138511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21006
I20260924 19:35:30.138640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:30.138787 21074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d: No bootstrap required, opened a new log
I20260924 19:35:30.138830 392 external_mini_cluster.cc:1428] Setting key 497f08f1181ba2ede0312ebab6ec65c4
I20260924 19:35:30.140597 21074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261fb9053f9d4272925cf94a06e10c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260924 19:35:30.140789 21074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:30.140887 21074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261fb9053f9d4272925cf94a06e10c8d, State: Initialized, Role: FOLLOWER
I20260924 19:35:30.141023 21074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261fb9053f9d4272925cf94a06e10c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260924 19:35:30.141099 21074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:30.141130 21074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:30.141175 21074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:30.141690 21074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261fb9053f9d4272925cf94a06e10c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260924 19:35:30.141772 21074 leader_election.cc:302] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261fb9053f9d4272925cf94a06e10c8d; no voters:
I20260924 19:35:30.141911 21074 leader_election.cc:288] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:30.141955 21079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:30.142143 21074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:30.142145 21079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [term 1 LEADER]: Becoming Leader. State: Replica: 261fb9053f9d4272925cf94a06e10c8d, State: Running, Role: LEADER
I20260924 19:35:30.142304 21079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261fb9053f9d4272925cf94a06e10c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } }
I20260924 19:35:30.142679 21080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261fb9053f9d4272925cf94a06e10c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261fb9053f9d4272925cf94a06e10c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } } }
I20260924 19:35:30.142755 21080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:30.142949 21081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261fb9053f9d4272925cf94a06e10c8d. Latest consensus state: current_term: 1 leader_uuid: "261fb9053f9d4272925cf94a06e10c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261fb9053f9d4272925cf94a06e10c8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40537 } } }
I20260924 19:35:30.143050 21081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:30.143381 21088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:30.144014 21088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:30.145766 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:30.139327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37535 (local address 127.0.0.1:40537)
0924 19:35:30.139489 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:30.139496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:30.139519 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:35:30.140569 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:30.140574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:30.140617 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:30.140774 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:30.141016 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:30.142465 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:30.143659 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:30.144230 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:30.144331 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:30.145098 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:30.145102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:30.145102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:30.145301 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:30.145339 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:30.145354 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:30.145418 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:30.145509 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:35:30.145565 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260924 19:35:30.146003 21088 catalog_manager.cc:1347] Generated new cluster ID: f260afc9ae08417193076fe67f1b938c
I20260924 19:35:30.146044 21088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:30.146606 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:30.151295 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:30.148156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33099 (local address 127.0.0.1:40537)
0924 19:35:30.148192 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:30.148194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:30.148253 (+ 59us) server_negotiation.cc:407] Connection header received
0924 19:35:30.148312 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:30.148312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:30.148322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:30.148367 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:30.148476 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:30.149173 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:30.150144 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:30.150411 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:30.150470 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:30.151094 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:30.151095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:30.151095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:30.151099 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:30.151113 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:30.151117 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:30.151150 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:30.151202 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 19:35:30.151223 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:35:30.156148 21088 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:30.156793 21088 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:30.161767 21088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261fb9053f9d4272925cf94a06e10c8d: Generated new TSK 0
I20260924 19:35:30.161931 21088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:30.167696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21006
2026-09-24T19:35:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:30 dist-test-slave-m8jb krb5kdc[21105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:30 dist-test-slave-m8jb krb5kdc[21105](info): set up 2 sockets
Sep 24 19:35:30 dist-test-slave-m8jb krb5kdc[21105](info): commencing operation
krb5kdc: starting...
W20260924 19:35:31.108390 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:31 dist-test-slave-m8jb krb5kdc[21105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:31.220767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:31.299825 21118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:31.300032 21118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:31.300063 21118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:31.301281 21118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:31.301322 21118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:31.301342 21118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:31.301357 21118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:31.301373 21118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:31.302881 21118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:31.303131 21118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:31.303339 21118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:31.308262 21124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:31.308279 21126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:31.308373 21118 server_base.cc:1034] running on GCE node
W20260924 19:35:31.308287 21123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:31.308719 21118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:31.309005 21118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:31.310155 21118 hybrid_clock.cc:648] HybridClock initialized: now 1790278531310139 us; error 31 us; skew 500 ppm
Sep 24 19:35:31 dist-test-slave-m8jb krb5kdc[21105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:31.312883 21118 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:31.313309 21118 webserver.cc:460] Webserver started at http://127.0.0.1:36351/ using document root <none> and password file <none>
I20260924 19:35:31.313526 21118 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:31.313575 21118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:31.313709 21118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:31.314816 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "92632cd1b50b45ef8c1435951ed7e5ad"
format_stamp: "Formatted at 2026-09-24 19:35:31 on dist-test-slave-m8jb"
server_key: "e37ae27d3aade4fabb47fa24b1835822"
server_key_iv: "7f9926c5c2ac00c388073b096c49a68a"
server_key_version: "encryptionkey@0"
I20260924 19:35:31.315132 21118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "92632cd1b50b45ef8c1435951ed7e5ad"
format_stamp: "Formatted at 2026-09-24 19:35:31 on dist-test-slave-m8jb"
server_key: "e37ae27d3aade4fabb47fa24b1835822"
server_key_iv: "7f9926c5c2ac00c388073b096c49a68a"
server_key_version: "encryptionkey@0"
I20260924 19:35:31.316411 21118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:31.317227 21133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:31.317399 21118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:31.317482 21118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "92632cd1b50b45ef8c1435951ed7e5ad"
format_stamp: "Formatted at 2026-09-24 19:35:31 on dist-test-slave-m8jb"
server_key: "e37ae27d3aade4fabb47fa24b1835822"
server_key_iv: "7f9926c5c2ac00c388073b096c49a68a"
server_key_version: "encryptionkey@0"
I20260924 19:35:31.317545 21118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:31.320840 21118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:31.322888 21118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:31.323019 21118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:31.326699 21118 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36127
I20260924 19:35:31.326740 21185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36127 every 8 connection(s)
I20260924 19:35:31.327041 21118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:31.328024 21186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:31.330269 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad: Bootstrap starting.
I20260924 19:35:31.330821 21186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:31.331071 21186 log.cc:824] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:31.331772 21186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad: No bootstrap required, opened a new log
I20260924 19:35:31.333402 21186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92632cd1b50b45ef8c1435951ed7e5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } }
I20260924 19:35:31.333551 21186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:31.333575 21186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92632cd1b50b45ef8c1435951ed7e5ad, State: Initialized, Role: FOLLOWER
I20260924 19:35:31.333700 21186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92632cd1b50b45ef8c1435951ed7e5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } }
I20260924 19:35:31.333771 21186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:31.333792 21186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:31.333822 21186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:31.334335 21186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92632cd1b50b45ef8c1435951ed7e5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } }
I20260924 19:35:31.334420 21186 leader_election.cc:302] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92632cd1b50b45ef8c1435951ed7e5ad; no voters:
I20260924 19:35:31.334589 21186 leader_election.cc:288] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:31.334684 21189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:31.334815 21186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:31.334838 21189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [term 1 LEADER]: Becoming Leader. State: Replica: 92632cd1b50b45ef8c1435951ed7e5ad, State: Running, Role: LEADER
I20260924 19:35:31.335100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21118
I20260924 19:35:31.335201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:31.335379 392 external_mini_cluster.cc:1428] Setting key c950c8571087ced0916dd00e9ba97208
I20260924 19:35:31.335582 21189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92632cd1b50b45ef8c1435951ed7e5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } }
I20260924 19:35:31.336175 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92632cd1b50b45ef8c1435951ed7e5ad. Latest consensus state: current_term: 1 leader_uuid: "92632cd1b50b45ef8c1435951ed7e5ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92632cd1b50b45ef8c1435951ed7e5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } } }
I20260924 19:35:31.336156 21190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92632cd1b50b45ef8c1435951ed7e5ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92632cd1b50b45ef8c1435951ed7e5ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36127 } } }
I20260924 19:35:31.336315 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:31.336321 21190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:31.336520 21206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:31.337057 21206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:31.337996 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:31.336037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38823 (local address 127.0.0.1:36127)
0924 19:35:31.336431 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:31.336441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:31.336479 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:35:31.336619 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:31.336626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:31.336680 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:31.336871 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:31.337498 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:31.337501 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:31.337502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:31.337507 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:31.337550 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:31.337566 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:31.337634 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:31.337789 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:35:31.337837 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":78,"threads_started":1}
I20260924 19:35:31.338703 21206 catalog_manager.cc:1347] Generated new cluster ID: ac71c78afb1c47c79fe737fb57f659a8
I20260924 19:35:31.338791 21206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:31.338851 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:31.341707 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:31.340165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34317 (local address 127.0.0.1:36127)
0924 19:35:31.340199 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:31.340200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:31.340852 (+ 652us) server_negotiation.cc:407] Connection header received
0924 19:35:31.340905 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:31.340906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:31.340912 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:31.340962 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:31.341479 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:31.341480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:31.341480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:31.341484 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:31.341498 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:31.341502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:31.341547 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:31.341629 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 19:35:31.341643 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:31.345368 21206 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:31.345515 21206 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:31.349350 21206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92632cd1b50b45ef8c1435951ed7e5ad: Generated new TSK 0
I20260924 19:35:31.349510 21206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:31.352563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21118
2026-09-24T19:35:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:31 dist-test-slave-m8jb krb5kdc[21217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:31 dist-test-slave-m8jb krb5kdc[21217](info): set up 2 sockets
Sep 24 19:35:31 dist-test-slave-m8jb krb5kdc[21217](info): commencing operation
krb5kdc: starting...
W20260924 19:35:32.304410 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:32 dist-test-slave-m8jb krb5kdc[21217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:32.415189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:32.493302 21230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:32.493521 21230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:32.493552 21230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:32.494741 21230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:32.494787 21230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:32.494807 21230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:32.494822 21230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:32.494838 21230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:32.496316 21230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:32.496563 21230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:32.496791 21230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:32.501363 21235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:32.501381 21238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:32.501372 21236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:32.501626 21230 server_base.cc:1034] running on GCE node
I20260924 19:35:32.501843 21230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:32.502140 21230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:32.503283 21230 hybrid_clock.cc:648] HybridClock initialized: now 1790278532503270 us; error 29 us; skew 500 ppm
Sep 24 19:35:32 dist-test-slave-m8jb krb5kdc[21217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:32.505620 21230 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:32.506006 21230 webserver.cc:460] Webserver started at http://127.0.0.1:39175/ using document root <none> and password file <none>
I20260924 19:35:32.506243 21230 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:32.506289 21230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:32.506410 21230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:32.507427 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c"
format_stamp: "Formatted at 2026-09-24 19:35:32 on dist-test-slave-m8jb"
server_key: "73a98792e3f6ce322e8fe10b5e97bb4d"
server_key_iv: "3cc597b230d224d7102e92b8a30a9e78"
server_key_version: "encryptionkey@0"
I20260924 19:35:32.507745 21230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c"
format_stamp: "Formatted at 2026-09-24 19:35:32 on dist-test-slave-m8jb"
server_key: "73a98792e3f6ce322e8fe10b5e97bb4d"
server_key_iv: "3cc597b230d224d7102e92b8a30a9e78"
server_key_version: "encryptionkey@0"
I20260924 19:35:32.508975 21230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:35:32.509692 21245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:32.509858 21230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:32.509938 21230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c"
format_stamp: "Formatted at 2026-09-24 19:35:32 on dist-test-slave-m8jb"
server_key: "73a98792e3f6ce322e8fe10b5e97bb4d"
server_key_iv: "3cc597b230d224d7102e92b8a30a9e78"
server_key_version: "encryptionkey@0"
I20260924 19:35:32.509996 21230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:32.530148 21230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:32.532111 21230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:32.532245 21230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:32.536032 21230 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40281
I20260924 19:35:32.536085 21297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40281 every 8 connection(s)
I20260924 19:35:32.536402 21230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:32.537220 21298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:32.539433 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c: Bootstrap starting.
I20260924 19:35:32.539726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21230
I20260924 19:35:32.539829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:32.539970 392 external_mini_cluster.cc:1428] Setting key 5983adb8c9dce41804a5cb2174bd9167
I20260924 19:35:32.540038 21298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:32.540741 21298 log.cc:824] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:32.541378 21298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c: No bootstrap required, opened a new log
I20260924 19:35:32.543467 21298 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260924 19:35:32.543643 21298 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:32.543675 21298 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6cd9b0baeb5442d9bc03a0e94dca14c, State: Initialized, Role: FOLLOWER
I20260924 19:35:32.543790 21298 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260924 19:35:32.543855 21298 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:32.543885 21298 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:32.543926 21298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:32.544449 21298 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260924 19:35:32.544538 21298 leader_election.cc:302] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6cd9b0baeb5442d9bc03a0e94dca14c; no voters:
I20260924 19:35:32.544735 21298 leader_election.cc:288] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:32.544775 21303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:32.544903 21303 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [term 1 LEADER]: Becoming Leader. State: Replica: b6cd9b0baeb5442d9bc03a0e94dca14c, State: Running, Role: LEADER
I20260924 19:35:32.544981 21303 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } }
I20260924 19:35:32.544982 21298 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:32.545310 21303 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6cd9b0baeb5442d9bc03a0e94dca14c. Latest consensus state: current_term: 1 leader_uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260924 19:35:32.545431 21303 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:32.545899 21310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:32.546350 21310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 19:35:32 dist-test-slave-m8jb krb5kdc[21217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:35:32.546600 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6cd9b0baeb5442d9bc03a0e94dca14c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40281 } } }
I20260924 19:35:32.547747 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:32.548393 21310 catalog_manager.cc:1347] Generated new cluster ID: 745206230c5c4422939e82cd4cac60a5
I20260924 19:35:32.548454 21310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:32.551139 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:32.540458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:57467 (local address 127.0.0.1:40281)
0924 19:35:32.540662 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:32.540672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:32.540704 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:35:32.541275 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:32.541283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:32.541336 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:32.541467 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:32.542189 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:32.543075 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:32.544099 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:32.544487 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:32.544583 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:32.547671 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:32.547675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:32.547676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:32.547688 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:32.549553 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:32.549983 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:32.549986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:32.549986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:32.550088 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:32.550297 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:32.550297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:32.550298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:32.550566 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:32.550704 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:32.550839 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 19:35:32.550926 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260924 19:35:32.551951 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:32.559649 21310 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:32.560221 21301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:32.553364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:49407 (local address 127.0.0.1:40281)
0924 19:35:32.553395 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:32.553397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:32.553484 (+ 87us) server_negotiation.cc:407] Connection header received
0924 19:35:32.553530 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:32.553530 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:32.553539 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:32.553586 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:32.554194 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:32.554919 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:32.555876 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:32.556187 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:32.556238 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:32.557578 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:32.557579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:32.557579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:32.557585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:32.559048 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:32.559417 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:32.559418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:32.559418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:32.559473 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:32.559704 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:32.559705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:32.559705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:32.559833 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:32.559936 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:32.560079 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 19:35:32.560138 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:35:32.560491 21310 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:32.570103 21310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6cd9b0baeb5442d9bc03a0e94dca14c: Generated new TSK 0
I20260924 19:35:32.570281 21310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:32.574640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21230
W20260924 19:35:32.578413 21215 connection.cc:535] client connection to 127.0.0.1:40281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40281): Connection reset by peer (error 104)
2026-09-24T19:35:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:32 dist-test-slave-m8jb krb5kdc[21329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:32 dist-test-slave-m8jb krb5kdc[21329](info): set up 2 sockets
Sep 24 19:35:32 dist-test-slave-m8jb krb5kdc[21329](info): commencing operation
krb5kdc: starting...
W20260924 19:35:33.524446 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:33 dist-test-slave-m8jb krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:33.644536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:33.724772 21342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:33.724963 21342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:33.724985 21342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:33.726114 21342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:33.726154 21342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:33.726167 21342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:33.726177 21342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:33.726189 21342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:33.727696 21342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:33.727926 21342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:33.728135 21342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:33.732703 21348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:33.732697 21347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:33.732697 21350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:33.733007 21342 server_base.cc:1034] running on GCE node
I20260924 19:35:33.733245 21342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:33.733536 21342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:33.734673 21342 hybrid_clock.cc:648] HybridClock initialized: now 1790278533734662 us; error 33 us; skew 500 ppm
Sep 24 19:35:33 dist-test-slave-m8jb krb5kdc[21329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278533, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:33.743839 21342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:33.744292 21342 webserver.cc:460] Webserver started at http://127.0.0.1:39879/ using document root <none> and password file <none>
I20260924 19:35:33.744516 21342 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:33.744561 21342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:33.744716 21342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:33.745743 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "a40e011f4da14190bb67c848598faef5"
format_stamp: "Formatted at 2026-09-24 19:35:33 on dist-test-slave-m8jb"
server_key: "50333aced191a23ccbaf51cbe05c3479"
server_key_iv: "ba2c0d981b4bef1860b28cebee7ac1eb"
server_key_version: "encryptionkey@0"
I20260924 19:35:33.746053 21342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "a40e011f4da14190bb67c848598faef5"
format_stamp: "Formatted at 2026-09-24 19:35:33 on dist-test-slave-m8jb"
server_key: "50333aced191a23ccbaf51cbe05c3479"
server_key_iv: "ba2c0d981b4bef1860b28cebee7ac1eb"
server_key_version: "encryptionkey@0"
I20260924 19:35:33.747324 21342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:33.748204 21357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:33.748373 21342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:33.748450 21342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "a40e011f4da14190bb67c848598faef5"
format_stamp: "Formatted at 2026-09-24 19:35:33 on dist-test-slave-m8jb"
server_key: "50333aced191a23ccbaf51cbe05c3479"
server_key_iv: "ba2c0d981b4bef1860b28cebee7ac1eb"
server_key_version: "encryptionkey@0"
I20260924 19:35:33.748529 21342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:33.765998 21342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:33.768175 21342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:33.768292 21342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:33.772251 21342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46229
I20260924 19:35:33.772301 21409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46229 every 8 connection(s)
I20260924 19:35:33.772624 21342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:33.773473 21410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:33.775672 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5: Bootstrap starting.
I20260924 19:35:33.776217 21410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:33.776470 21410 log.cc:824] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:33.777122 21410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5: No bootstrap required, opened a new log
I20260924 19:35:33.778712 21410 raft_consensus.cc:348] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40e011f4da14190bb67c848598faef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46229 } }
I20260924 19:35:33.778846 21410 raft_consensus.cc:374] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:33.778869 21410 raft_consensus.cc:729] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a40e011f4da14190bb67c848598faef5, State: Initialized, Role: FOLLOWER
I20260924 19:35:33.778996 21410 consensus_queue.cc:260] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40e011f4da14190bb67c848598faef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46229 } }
I20260924 19:35:33.779057 21410 raft_consensus.cc:388] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:33.779078 21410 raft_consensus.cc:482] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:33.779122 21410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:33.779394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21342
I20260924 19:35:33.779510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:33.779703 392 external_mini_cluster.cc:1428] Setting key 7a1910e4fbbb8816e1857be1ca761e53
I20260924 19:35:33.779794 21410 raft_consensus.cc:504] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40e011f4da14190bb67c848598faef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46229 } }
I20260924 19:35:33.779897 21410 leader_election.cc:302] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a40e011f4da14190bb67c848598faef5; no voters:
I20260924 19:35:33.780030 21410 leader_election.cc:288] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:33.780140 21413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:33.780236 21413 raft_consensus.cc:686] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [term 1 LEADER]: Becoming Leader. State: Replica: a40e011f4da14190bb67c848598faef5, State: Running, Role: LEADER
I20260924 19:35:33.780272 21410 sys_catalog.cc:564] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:33.780318 21413 consensus_queue.cc:237] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40e011f4da14190bb67c848598faef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46229 } }
I20260924 19:35:33.780647 21416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a40e011f4da14190bb67c848598faef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40e011f4da14190bb67c848598faef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46229 } } }
I20260924 19:35:33.780731 21416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:33.780618 21413 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a40e011f4da14190bb67c848598faef5. Latest consensus state: current_term: 1 leader_uuid: "a40e011f4da14190bb67c848598faef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a40e011f4da14190bb67c848598faef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46229 } } }
I20260924 19:35:33.780893 21413 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5 [sys.catalog]: This master's current role is: LEADER
W20260924 19:35:33.782392 21431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:33.782436 21431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:33.782485 21419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:33.782620 21419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:33.785691 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:33.780198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:42257 (local address 127.0.0.1:46229)
0924 19:35:33.780524 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:33.780533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:33.780586 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:35:33.780723 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:33.780729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:33.780787 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:33.781164 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:33.781322 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:33.782456 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:33.783531 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:33.784106 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:33.784214 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:33.784948 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:33.784952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:33.784952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:33.785165 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:33.785202 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:33.785216 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:33.785303 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:33.785428 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 19:35:33.785454 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":74,"threads_started":1}
I20260924 19:35:33.786670 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:33.790930 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:33.787966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:53581 (local address 127.0.0.1:46229)
0924 19:35:33.788021 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:33.788023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:33.788073 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:35:33.788139 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:33.788140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:33.788148 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:33.788191 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:33.788322 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:33.788849 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:33.789819 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:33.790041 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:33.790085 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:33.790699 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:33.790700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:33.790700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:33.790704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:33.790719 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:33.790723 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:33.790774 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:33.790850 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 19:35:33.790863 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:35:33.791309 21419 catalog_manager.cc:1347] Generated new cluster ID: ea01f709b5364c37af3cf9e4cb29798f
I20260924 19:35:33.791365 21419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:33.806362 21419 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:33.806913 21419 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:33.811848 21419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a40e011f4da14190bb67c848598faef5: Generated new TSK 0
I20260924 19:35:33.812038 21419 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:33.817399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21342
W20260924 19:35:33.821280 21327 connection.cc:535] client connection to 127.0.0.1:46229 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46229): Connection reset by peer (error 104)
2026-09-24T19:35:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1242 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:33 dist-test-slave-m8jb krb5kdc[21441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:33 dist-test-slave-m8jb krb5kdc[21441](info): set up 2 sockets
Sep 24 19:35:33 dist-test-slave-m8jb krb5kdc[21441](info): commencing operation
krb5kdc: starting...
W20260924 19:35:34.764494 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:34 dist-test-slave-m8jb krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:34.879921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:34.959734 21454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:34.959939 21454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:34.959959 21454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:34.961102 21454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:34.961139 21454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:34.961153 21454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:34.961163 21454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:34.961174 21454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:34.962590 21454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:34.962770 21454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:34.962939 21454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:34.967721 21460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:34.967715 21459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:34.967715 21462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:34.967986 21454 server_base.cc:1034] running on GCE node
I20260924 19:35:34.968200 21454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:34.968524 21454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:34.969688 21454 hybrid_clock.cc:648] HybridClock initialized: now 1790278534969681 us; error 47 us; skew 500 ppm
Sep 24 19:35:34 dist-test-slave-m8jb krb5kdc[21441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:34.972090 21454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:34.972548 21454 webserver.cc:460] Webserver started at http://127.0.0.1:44781/ using document root <none> and password file <none>
I20260924 19:35:34.972792 21454 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:34.972849 21454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:34.972970 21454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:34.974036 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "f2bf113604904348b9400ef8b7d1446c"
format_stamp: "Formatted at 2026-09-24 19:35:34 on dist-test-slave-m8jb"
server_key: "00a0611e12e0203f68dba53a8ffc7b1a"
server_key_iv: "e262f53210528f8e610366cac1e29999"
server_key_version: "encryptionkey@0"
I20260924 19:35:34.974372 21454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2bf113604904348b9400ef8b7d1446c"
format_stamp: "Formatted at 2026-09-24 19:35:34 on dist-test-slave-m8jb"
server_key: "00a0611e12e0203f68dba53a8ffc7b1a"
server_key_iv: "e262f53210528f8e610366cac1e29999"
server_key_version: "encryptionkey@0"
I20260924 19:35:34.975597 21454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:34.976302 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:34.976477 21454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:34.976557 21454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "f2bf113604904348b9400ef8b7d1446c"
format_stamp: "Formatted at 2026-09-24 19:35:34 on dist-test-slave-m8jb"
server_key: "00a0611e12e0203f68dba53a8ffc7b1a"
server_key_iv: "e262f53210528f8e610366cac1e29999"
server_key_version: "encryptionkey@0"
I20260924 19:35:34.976619 21454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:34.979809 21454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:34.981582 21454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:34.981703 21454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:34.985351 21454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40845
I20260924 19:35:34.985383 21521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40845 every 8 connection(s)
I20260924 19:35:34.985668 21454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:34.986590 21522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:34.988874 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c: Bootstrap starting.
I20260924 19:35:34.989415 21522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:34.989666 21522 log.cc:824] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:34.990342 21522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c: No bootstrap required, opened a new log
I20260924 19:35:34.991923 21522 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bf113604904348b9400ef8b7d1446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40845 } }
I20260924 19:35:34.992065 21522 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:34.992086 21522 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2bf113604904348b9400ef8b7d1446c, State: Initialized, Role: FOLLOWER
I20260924 19:35:34.992213 21522 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bf113604904348b9400ef8b7d1446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40845 } }
I20260924 19:35:34.992273 21522 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:34.992293 21522 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:34.992324 21522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:34.992914 21522 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bf113604904348b9400ef8b7d1446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40845 } }
I20260924 19:35:34.993006 21522 leader_election.cc:302] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2bf113604904348b9400ef8b7d1446c; no voters:
I20260924 19:35:34.993165 21522 leader_election.cc:288] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:34.993203 21525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:34.993343 21525 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [term 1 LEADER]: Becoming Leader. State: Replica: f2bf113604904348b9400ef8b7d1446c, State: Running, Role: LEADER
I20260924 19:35:34.993418 21522 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:34.993476 21525 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bf113604904348b9400ef8b7d1446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40845 } }
I20260924 19:35:34.993819 21526 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2bf113604904348b9400ef8b7d1446c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bf113604904348b9400ef8b7d1446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40845 } } }
I20260924 19:35:34.993919 21526 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:34.993832 21527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2bf113604904348b9400ef8b7d1446c. Latest consensus state: current_term: 1 leader_uuid: "f2bf113604904348b9400ef8b7d1446c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2bf113604904348b9400ef8b7d1446c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40845 } } }
I20260924 19:35:34.993996 21527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:34.994146 21529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:34.994419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21454
I20260924 19:35:34.994532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:34.994709 21529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:34.994740 392 external_mini_cluster.cc:1428] Setting key 2a8a4b3438ca0a1542f18f10a5d65130
I20260924 19:35:34.996295 21529 catalog_manager.cc:1347] Generated new cluster ID: cbe90603f2784b4182d91396bc3ccd56
I20260924 19:35:34.996341 21529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:34.997068 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:34.995179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:41427 (local address 127.0.0.1:40845)
0924 19:35:34.995514 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:34.995522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:34.995553 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:35:34.995664 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:34.995671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:34.995701 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:34.995844 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:34.996604 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:34.996607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:34.996607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:34.996612 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:34.996714 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:34.996729 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:34.996795 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:34.996850 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 19:35:34.996904 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":68,"threads_started":1}
I20260924 19:35:34.997880 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:35.000423 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:34.999363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:32793 (local address 127.0.0.1:40845)
0924 19:35:34.999401 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:34.999403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:34.999560 (+ 157us) server_negotiation.cc:407] Connection header received
0924 19:35:34.999624 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:34.999625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:34.999639 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:34.999681 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:35.000236 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:35.000237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:35.000238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:35.000241 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:35.000255 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:35.000259 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:35.000295 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:35.000348 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:35:35.000361 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:35:35.007848 21529 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:35.008023 21529 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:35.016275 21529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2bf113604904348b9400ef8b7d1446c: Generated new TSK 0
I20260924 19:35:35.016477 21529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:35.020931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21454
2026-09-24T19:35:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:35 dist-test-slave-m8jb krb5kdc[21553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:35 dist-test-slave-m8jb krb5kdc[21553](info): set up 2 sockets
Sep 24 19:35:35 dist-test-slave-m8jb krb5kdc[21553](info): commencing operation
krb5kdc: starting...
W20260924 19:35:35.972468 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:36 dist-test-slave-m8jb krb5kdc[21553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:36.083973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:36.162025 21566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:36.162232 21566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:36.162262 21566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:36.163396 21566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:36.163436 21566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:36.163455 21566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:36.163470 21566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:36.163486 21566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:36.164973 21566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:36.165200 21566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:36.165402 21566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:36.170073 21574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:36.170152 21572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:36.170086 21571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:36.170408 21566 server_base.cc:1034] running on GCE node
I20260924 19:35:36.170610 21566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:36.170840 21566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:36.171980 21566 hybrid_clock.cc:648] HybridClock initialized: now 1790278536171964 us; error 30 us; skew 500 ppm
Sep 24 19:35:36 dist-test-slave-m8jb krb5kdc[21553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:36.174500 21566 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:36.174935 21566 webserver.cc:460] Webserver started at http://127.0.0.1:42747/ using document root <none> and password file <none>
I20260924 19:35:36.175128 21566 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:36.175171 21566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:36.175273 21566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:36.176317 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "de6b1d2a5aca459aa97fb5919e181b14"
format_stamp: "Formatted at 2026-09-24 19:35:36 on dist-test-slave-m8jb"
server_key: "4f5497ccaf184f022350549e733fbdd7"
server_key_iv: "0fc28f09a6e0153ce50297ab98fd3bdb"
server_key_version: "encryptionkey@0"
I20260924 19:35:36.176662 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "de6b1d2a5aca459aa97fb5919e181b14"
format_stamp: "Formatted at 2026-09-24 19:35:36 on dist-test-slave-m8jb"
server_key: "4f5497ccaf184f022350549e733fbdd7"
server_key_iv: "0fc28f09a6e0153ce50297ab98fd3bdb"
server_key_version: "encryptionkey@0"
I20260924 19:35:36.177881 21566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:35:36.178522 21581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:36.178645 21566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:36.178720 21566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "de6b1d2a5aca459aa97fb5919e181b14"
format_stamp: "Formatted at 2026-09-24 19:35:36 on dist-test-slave-m8jb"
server_key: "4f5497ccaf184f022350549e733fbdd7"
server_key_iv: "0fc28f09a6e0153ce50297ab98fd3bdb"
server_key_version: "encryptionkey@0"
I20260924 19:35:36.178791 21566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:36.200584 21566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:36.202745 21566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:36.202857 21566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:36.206588 21566 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41525
I20260924 19:35:36.206635 21633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41525 every 8 connection(s)
I20260924 19:35:36.206948 21566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:36.207765 21634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:36.208802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21566
I20260924 19:35:36.208932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:36.209129 392 external_mini_cluster.cc:1428] Setting key 657ebde685326528097a7eb4591597fd
I20260924 19:35:36.210377 21634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14: Bootstrap starting.
I20260924 19:35:36.211118 21634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:36.211376 21634 log.cc:824] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:36.212003 21634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14: No bootstrap required, opened a new log
I20260924 19:35:36.214365 21634 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6b1d2a5aca459aa97fb5919e181b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41525 } }
I20260924 19:35:36.214545 21634 raft_consensus.cc:374] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:36.214577 21634 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6b1d2a5aca459aa97fb5919e181b14, State: Initialized, Role: FOLLOWER
I20260924 19:35:36.214699 21634 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6b1d2a5aca459aa97fb5919e181b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41525 } }
I20260924 19:35:36.214776 21634 raft_consensus.cc:388] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:36.214807 21634 raft_consensus.cc:482] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:36.214847 21634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:36.215538 21634 raft_consensus.cc:504] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6b1d2a5aca459aa97fb5919e181b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41525 } }
I20260924 19:35:36.215626 21634 leader_election.cc:302] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de6b1d2a5aca459aa97fb5919e181b14; no voters:
I20260924 19:35:36.215773 21634 leader_election.cc:288] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:36.215819 21639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 1 FOLLOWER]: Leader election won for term 1
Sep 24 19:35:36 dist-test-slave-m8jb krb5kdc[21553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:35:36.215991 21639 raft_consensus.cc:686] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [term 1 LEADER]: Becoming Leader. State: Replica: de6b1d2a5aca459aa97fb5919e181b14, State: Running, Role: LEADER
I20260924 19:35:36.216044 21634 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:36.216126 21639 consensus_queue.cc:237] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6b1d2a5aca459aa97fb5919e181b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41525 } }
I20260924 19:35:36.216471 21639 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de6b1d2a5aca459aa97fb5919e181b14. Latest consensus state: current_term: 1 leader_uuid: "de6b1d2a5aca459aa97fb5919e181b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6b1d2a5aca459aa97fb5919e181b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41525 } } }
I20260924 19:35:36.216699 21641 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de6b1d2a5aca459aa97fb5919e181b14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de6b1d2a5aca459aa97fb5919e181b14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41525 } } }
I20260924 19:35:36.216768 21641 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:36.216976 21648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:36.217471 21648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:36.216547 21639 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:36.219041 21648 catalog_manager.cc:1347] Generated new cluster ID: fe9d6591d717449b88343459f0db9893
I20260924 19:35:36.219089 21648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:36.220252 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:36.209643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:48725 (local address 127.0.0.1:41525)
0924 19:35:36.209789 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:36.209798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:36.209831 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:35:36.210235 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:36.210241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:36.210286 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:36.210395 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:36.211118 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:36.212172 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:36.213260 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:36.213696 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:36.213785 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:36.216519 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:36.216523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:36.216524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:36.216537 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:36.218350 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:36.218774 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:36.218777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:36.218778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:36.218883 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:36.219117 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:36.219118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:36.219118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:36.219397 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:36.219533 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:36.219959 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 19:35:36.220048 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 19:35:36.221199 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:36.226598 21648 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:36.227124 21648 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:36.231289 21648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de6b1d2a5aca459aa97fb5919e181b14: Generated new TSK 0
I20260924 19:35:36.231463 21648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:36.233070 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:36.225001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:56063 (local address 127.0.0.1:41525)
0924 19:35:36.225043 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:36.225045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:36.226296 (+ 1251us) server_negotiation.cc:407] Connection header received
0924 19:35:36.226333 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:36.226334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:36.226345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:36.226388 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:36.227045 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:36.227745 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:36.228796 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:36.229111 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:36.229164 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:36.230476 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:36.230477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:36.230477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:36.230484 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:36.231977 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:36.232359 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:36.232360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:36.232360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:36.232407 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:36.232620 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:36.232621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:36.232621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:36.232756 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:36.232829 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:36.232969 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 19:35:36.233016 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:35:36.234970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21566
W20260924 19:35:36.238809 21551 connection.cc:535] client connection to 127.0.0.1:41525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41525): Connection reset by peer (error 104)
2026-09-24T19:35:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:36 dist-test-slave-m8jb krb5kdc[21665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:36 dist-test-slave-m8jb krb5kdc[21665](info): set up 2 sockets
Sep 24 19:35:36 dist-test-slave-m8jb krb5kdc[21665](info): commencing operation
krb5kdc: starting...
W20260924 19:35:37.184460 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:37 dist-test-slave-m8jb krb5kdc[21665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:37.297921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:37.376183 21678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:37.376403 21678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:37.376434 21678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:37.377671 21678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:37.377715 21678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:37.377733 21678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:37.377749 21678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:37.377765 21678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:37.379251 21678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:37.379506 21678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:37.379714 21678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:37.384367 21684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:37.384445 21683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:37.384541 21686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:37.384688 21678 server_base.cc:1034] running on GCE node
I20260924 19:35:37.384910 21678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:37.385150 21678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:37.386291 21678 hybrid_clock.cc:648] HybridClock initialized: now 1790278537386280 us; error 29 us; skew 500 ppm
Sep 24 19:35:37 dist-test-slave-m8jb krb5kdc[21665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:37.388537 21678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:37.388995 21678 webserver.cc:460] Webserver started at http://127.0.0.1:38031/ using document root <none> and password file <none>
I20260924 19:35:37.389210 21678 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:37.389252 21678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:37.389372 21678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:37.390383 21678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "279cd7efa6874b7190e112a9b6154d9a"
format_stamp: "Formatted at 2026-09-24 19:35:37 on dist-test-slave-m8jb"
server_key: "597460de6e385f6805b2775f17074033"
server_key_iv: "de3e74db2b6a6d83aef183e187a4df0e"
server_key_version: "encryptionkey@0"
I20260924 19:35:37.390697 21678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "279cd7efa6874b7190e112a9b6154d9a"
format_stamp: "Formatted at 2026-09-24 19:35:37 on dist-test-slave-m8jb"
server_key: "597460de6e385f6805b2775f17074033"
server_key_iv: "de3e74db2b6a6d83aef183e187a4df0e"
server_key_version: "encryptionkey@0"
I20260924 19:35:37.391917 21678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 19:35:37.392606 21693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:37.392757 21678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:37.392834 21678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "279cd7efa6874b7190e112a9b6154d9a"
format_stamp: "Formatted at 2026-09-24 19:35:37 on dist-test-slave-m8jb"
server_key: "597460de6e385f6805b2775f17074033"
server_key_iv: "de3e74db2b6a6d83aef183e187a4df0e"
server_key_version: "encryptionkey@0"
I20260924 19:35:37.392900 21678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:37.414780 21678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:37.416759 21678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:37.416882 21678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:37.420581 21678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42989
I20260924 19:35:37.420622 21745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42989 every 8 connection(s)
I20260924 19:35:37.420966 21678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:37.421860 21746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:37.422452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21678
I20260924 19:35:37.422562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:37.422734 392 external_mini_cluster.cc:1428] Setting key 735e4af4441275422f985d753d2d6a19
I20260924 19:35:37.424801 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a: Bootstrap starting.
I20260924 19:35:37.425467 21746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:37.425767 21746 log.cc:824] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:37.426385 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a: No bootstrap required, opened a new log
I20260924 19:35:37.428416 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:37.423215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:35351 (local address 127.0.0.1:42989)
0924 19:35:37.423405 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:37.423414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:37.423448 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:35:37.423965 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:37.423974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:37.424040 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:37.424181 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:37.424356 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:37.425662 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:37.426436 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:37.427033 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:37.427150 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:37.427749 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:37.427754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:37.427754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:37.427954 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:37.427996 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:37.428013 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:37.428105 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:37.428169 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 19:35:37.428197 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260924 19:35:37.428566 21746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279cd7efa6874b7190e112a9b6154d9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42989 } }
I20260924 19:35:37.428743 21746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:37.428771 21746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279cd7efa6874b7190e112a9b6154d9a, State: Initialized, Role: FOLLOWER
I20260924 19:35:37.428917 21746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279cd7efa6874b7190e112a9b6154d9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42989 } }
I20260924 19:35:37.428977 21746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:37.428999 21746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:37.429042 21746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:37.429490 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:37.429662 21746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279cd7efa6874b7190e112a9b6154d9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42989 } }
I20260924 19:35:37.429768 21746 leader_election.cc:302] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 279cd7efa6874b7190e112a9b6154d9a; no voters:
I20260924 19:35:37.429919 21746 leader_election.cc:288] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:37.429952 21751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:37.430117 21751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [term 1 LEADER]: Becoming Leader. State: Replica: 279cd7efa6874b7190e112a9b6154d9a, State: Running, Role: LEADER
I20260924 19:35:37.430145 21746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:37.430305 21751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279cd7efa6874b7190e112a9b6154d9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42989 } }
I20260924 19:35:37.430647 21752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "279cd7efa6874b7190e112a9b6154d9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279cd7efa6874b7190e112a9b6154d9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42989 } } }
I20260924 19:35:37.430734 21752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:37.430648 21753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 279cd7efa6874b7190e112a9b6154d9a. Latest consensus state: current_term: 1 leader_uuid: "279cd7efa6874b7190e112a9b6154d9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "279cd7efa6874b7190e112a9b6154d9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42989 } } }
I20260924 19:35:37.430927 21753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:37.430974 21763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:37.431470 21763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:37.432966 21763 catalog_manager.cc:1347] Generated new cluster ID: c8e79ad86b7a42358f9c2ee6f4a4921a
I20260924 19:35:37.433017 21763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:37.435142 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:37.431661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:60203 (local address 127.0.0.1:42989)
0924 19:35:37.431693 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:37.431695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:37.431902 (+ 207us) server_negotiation.cc:407] Connection header received
0924 19:35:37.431958 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:37.431958 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:37.431969 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:37.432018 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:37.432189 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:37.432821 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:37.433846 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:37.434131 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:37.434178 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:37.434835 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:37.434837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:37.434837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:37.434841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:37.434854 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:37.434857 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:37.434895 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:37.435068 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 19:35:37.435076 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:37.439877 21763 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:37.440369 21763 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:37.443024 21763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 279cd7efa6874b7190e112a9b6154d9a: Generated new TSK 0
I20260924 19:35:37.443200 21763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:37.452620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21678
W20260924 19:35:37.456301 21663 connection.cc:535] client connection to 127.0.0.1:42989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42989): Connection reset by peer (error 104)
2026-09-24T19:35:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:37 dist-test-slave-m8jb krb5kdc[21777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:37 dist-test-slave-m8jb krb5kdc[21777](info): set up 2 sockets
Sep 24 19:35:37 dist-test-slave-m8jb krb5kdc[21777](info): commencing operation
krb5kdc: starting...
W20260924 19:35:38.392518 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:38 dist-test-slave-m8jb krb5kdc[21777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:35:38.504264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:38.582233 21790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:38.582445 21790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:38.582476 21790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:38.583606 21790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:38.583647 21790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:38.583664 21790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:38.583679 21790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:38.583694 21790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:38.585137 21790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:38.585371 21790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:38.585614 21790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:38.590126 21795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:38.590165 21796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:38.590320 21798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:38.590307 21790 server_base.cc:1034] running on GCE node
I20260924 19:35:38.590607 21790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:38.590862 21790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:38.592010 21790 hybrid_clock.cc:648] HybridClock initialized: now 1790278538591991 us; error 28 us; skew 500 ppm
Sep 24 19:35:38 dist-test-slave-m8jb krb5kdc[21777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:38.594347 21790 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:35:38.594764 21790 webserver.cc:460] Webserver started at http://127.0.0.1:44645/ using document root <none> and password file <none>
I20260924 19:35:38.594957 21790 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:38.595002 21790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:38.595124 21790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:38.596140 21790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "f4b11836580d4fcaa81e8ea42afec5d5"
format_stamp: "Formatted at 2026-09-24 19:35:38 on dist-test-slave-m8jb"
server_key: "3e86b67161f3f6ecfd30492b39eabab9"
server_key_iv: "937409e35c5300d5dd7f7bdc9c8a6719"
server_key_version: "encryptionkey@0"
I20260924 19:35:38.596465 21790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4b11836580d4fcaa81e8ea42afec5d5"
format_stamp: "Formatted at 2026-09-24 19:35:38 on dist-test-slave-m8jb"
server_key: "3e86b67161f3f6ecfd30492b39eabab9"
server_key_iv: "937409e35c5300d5dd7f7bdc9c8a6719"
server_key_version: "encryptionkey@0"
I20260924 19:35:38.597743 21790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:38.598531 21805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:38.598675 21790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:38.598758 21790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "f4b11836580d4fcaa81e8ea42afec5d5"
format_stamp: "Formatted at 2026-09-24 19:35:38 on dist-test-slave-m8jb"
server_key: "3e86b67161f3f6ecfd30492b39eabab9"
server_key_iv: "937409e35c5300d5dd7f7bdc9c8a6719"
server_key_version: "encryptionkey@0"
I20260924 19:35:38.598824 21790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:38.602033 21790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:38.603971 21790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:38.604101 21790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:38.607843 21790 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37083
I20260924 19:35:38.607879 21857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37083 every 8 connection(s)
I20260924 19:35:38.608199 21790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:38.608520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21790
I20260924 19:35:38.608667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:38.608924 392 external_mini_cluster.cc:1428] Setting key 14ac9c5b4bd9dcc6d71a630113c09093
I20260924 19:35:38.609321 21858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:38.611182 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:38.609446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:32959 (local address 127.0.0.1:37083)
0924 19:35:38.609616 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:38.609623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:38.609647 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:35:38.610067 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:38.610074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:38.610098 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:38.610209 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:38.610795 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:38.610798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:38.610798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:38.610802 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:38.610837 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:38.610847 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:38.610897 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:38.610969 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 19:35:38.611002 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260924 19:35:38.612008 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:38.612447 21858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5: Bootstrap starting.
I20260924 19:35:38.613183 21858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:38.613510 21858 log.cc:824] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:38.614161 21858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5: No bootstrap required, opened a new log
I20260924 19:35:38.614629 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:38.613551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:42575 (local address 127.0.0.1:37083)
0924 19:35:38.613585 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:38.613587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:38.613644 (+ 57us) server_negotiation.cc:407] Connection header received
0924 19:35:38.613717 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:38.613717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:38.613721 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:38.613761 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:38.614302 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:38.614303 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:38.614303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:38.614307 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:38.614389 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:38.614399 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:38.614472 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:38.614538 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 19:35:38.614557 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 19:35:38.615932 21858 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b11836580d4fcaa81e8ea42afec5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260924 19:35:38.616065 21858 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:38.616096 21858 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4b11836580d4fcaa81e8ea42afec5d5, State: Initialized, Role: FOLLOWER
I20260924 19:35:38.616214 21858 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b11836580d4fcaa81e8ea42afec5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260924 19:35:38.616291 21858 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:38.616333 21858 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:38.616379 21858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:38.616928 21858 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b11836580d4fcaa81e8ea42afec5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260924 19:35:38.617024 21858 leader_election.cc:302] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4b11836580d4fcaa81e8ea42afec5d5; no voters:
I20260924 19:35:38.617177 21858 leader_election.cc:288] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:38.617223 21868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:38.617414 21858 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:38.617431 21868 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [term 1 LEADER]: Becoming Leader. State: Replica: f4b11836580d4fcaa81e8ea42afec5d5, State: Running, Role: LEADER
I20260924 19:35:38.618131 21868 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b11836580d4fcaa81e8ea42afec5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } }
I20260924 19:35:38.618721 21869 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4b11836580d4fcaa81e8ea42afec5d5. Latest consensus state: current_term: 1 leader_uuid: "f4b11836580d4fcaa81e8ea42afec5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b11836580d4fcaa81e8ea42afec5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } } }
I20260924 19:35:38.618815 21870 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4b11836580d4fcaa81e8ea42afec5d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4b11836580d4fcaa81e8ea42afec5d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37083 } } }
I20260924 19:35:38.618937 21869 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:38.618969 21870 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5 [sys.catalog]: This master's current role is: LEADER
W20260924 19:35:38.619076 21882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:38.619127 21882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:38.620781 21885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:38.620954 21885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:38.622538 21885 catalog_manager.cc:1347] Generated new cluster ID: 1c0fe6f996ab4918b468adccd876dd2a
I20260924 19:35:38.622581 21885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:38.631433 21885 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:38.631603 21885 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:38.640900 21885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4b11836580d4fcaa81e8ea42afec5d5: Generated new TSK 0
I20260924 19:35:38.641120 21885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:38.644454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21790
2026-09-24T19:35:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:38 dist-test-slave-m8jb krb5kdc[21889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:38 dist-test-slave-m8jb krb5kdc[21889](info): set up 2 sockets
Sep 24 19:35:38 dist-test-slave-m8jb krb5kdc[21889](info): commencing operation
krb5kdc: starting...
W20260924 19:35:39.592576 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:39 dist-test-slave-m8jb krb5kdc[21889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:39.716260 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40033
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:39.800614 21902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:39.800876 21902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:39.800909 21902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:39.802099 21902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:39.802143 21902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:39.802162 21902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:39.802178 21902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:39.802193 21902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:39.803695 21902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:39.803936 21902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:39.804142 21902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:39.809003 21907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:39.809222 21902 server_base.cc:1034] running on GCE node
W20260924 19:35:39.809015 21908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:39.809034 21910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:39.809616 21902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:39.809914 21902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:39.811060 21902 hybrid_clock.cc:648] HybridClock initialized: now 1790278539811045 us; error 32 us; skew 500 ppm
Sep 24 19:35:39 dist-test-slave-m8jb krb5kdc[21889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:39.817438 21902 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:39.817930 21902 webserver.cc:460] Webserver started at http://127.0.98.62:34343/ using document root <none> and password file <none>
I20260924 19:35:39.818125 21902 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:39.818172 21902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:39.818297 21902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:39.819334 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "01a259fbad5040cd8d341508379ded74"
format_stamp: "Formatted at 2026-09-24 19:35:39 on dist-test-slave-m8jb"
server_key: "a2d835deb0cd9f07f3d8ea9c278281e3"
server_key_iv: "3a158463ceda1f9d6d41c712d4e0ba20"
server_key_version: "encryptionkey@0"
I20260924 19:35:39.819651 21902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "01a259fbad5040cd8d341508379ded74"
format_stamp: "Formatted at 2026-09-24 19:35:39 on dist-test-slave-m8jb"
server_key: "a2d835deb0cd9f07f3d8ea9c278281e3"
server_key_iv: "3a158463ceda1f9d6d41c712d4e0ba20"
server_key_version: "encryptionkey@0"
I20260924 19:35:39.820955 21902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:39.821692 21917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:39.821849 21902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:39.821933 21902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "01a259fbad5040cd8d341508379ded74"
format_stamp: "Formatted at 2026-09-24 19:35:39 on dist-test-slave-m8jb"
server_key: "a2d835deb0cd9f07f3d8ea9c278281e3"
server_key_iv: "3a158463ceda1f9d6d41c712d4e0ba20"
server_key_version: "encryptionkey@0"
I20260924 19:35:39.822000 21902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:39.843005 21902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:39.844857 21902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:39.844992 21902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:39.848729 21902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40033
I20260924 19:35:39.848779 21969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40033 every 8 connection(s)
I20260924 19:35:39.849112 21902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:39.849896 21970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:39.851159 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 21902
I20260924 19:35:39.851246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:39.851409 392 external_mini_cluster.cc:1428] Setting key 88f21ff49ae7b52dd9f2c0b60da8abc9
I20260924 19:35:39.852219 21970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74: Bootstrap starting.
I20260924 19:35:39.852783 21970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:39.853029 21970 log.cc:824] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:39.853617 21970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74: No bootstrap required, opened a new log
I20260924 19:35:39.855276 21970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a259fbad5040cd8d341508379ded74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260924 19:35:39.855408 21970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:39.855438 21970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a259fbad5040cd8d341508379ded74, State: Initialized, Role: FOLLOWER
I20260924 19:35:39.855582 21970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a259fbad5040cd8d341508379ded74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260924 19:35:39.855648 21970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:39.855677 21970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:39.855719 21970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:39.856267 21970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a259fbad5040cd8d341508379ded74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
I20260924 19:35:39.856352 21970 leader_election.cc:302] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01a259fbad5040cd8d341508379ded74; no voters:
I20260924 19:35:39.856508 21970 leader_election.cc:288] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:39.856563 21975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:39.856755 21975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [term 1 LEADER]: Becoming Leader. State: Replica: 01a259fbad5040cd8d341508379ded74, State: Running, Role: LEADER
I20260924 19:35:39.856769 21970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:39.856884 21975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a259fbad5040cd8d341508379ded74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } }
Sep 24 19:35:39 dist-test-slave-m8jb krb5kdc[21889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:39.857487 21977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a259fbad5040cd8d341508379ded74. Latest consensus state: current_term: 1 leader_uuid: "01a259fbad5040cd8d341508379ded74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a259fbad5040cd8d341508379ded74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } } }
I20260924 19:35:39.857533 21976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01a259fbad5040cd8d341508379ded74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a259fbad5040cd8d341508379ded74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40033 } } }
I20260924 19:35:39.857576 21977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:39.857601 21976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:39.857764 21984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:39.858295 21984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:39.860399 21984 catalog_manager.cc:1347] Generated new cluster ID: 35ca7931faaf433cacb0de9291cbc074
I20260924 19:35:39.860478 21984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:39.862051 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:39.851895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50809 (local address 127.0.98.62:40033)
0924 19:35:39.852066 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:39.852074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:39.852099 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:35:39.852512 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:39.852519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:39.852564 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:39.852719 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:39.853333 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:39.854241 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:39.854946 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:39.855333 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:39.855422 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:39.857989 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:39.857993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:39.857994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:39.858007 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:39.860258 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:39.860719 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:39.860722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:39.860722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:39.860835 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:39.861095 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:39.861096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:39.861096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:39.861428 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:39.861608 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:39.861720 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 19:35:39.861843 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260924 19:35:39.863230 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:39.872133 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:39.864900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35367 (local address 127.0.98.62:40033)
0924 19:35:39.864939 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:39.864940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:39.865004 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:35:39.865096 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:39.865097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:39.865106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:39.865134 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:39.865864 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:39.866602 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:39.867637 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:39.867971 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:39.868012 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:39.869460 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:39.869461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:39.869462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:39.869468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:39.870851 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:39.871249 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:39.871250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:39.871251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:39.871304 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:39.871517 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:39.871517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:39.871518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:39.871639 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:39.871736 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:39.871960 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:35:39.872024 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 19:35:39.912688 21984 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:39.913198 21984 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:39.917083 21984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01a259fbad5040cd8d341508379ded74: Generated new TSK 0
I20260924 19:35:39.917263 21984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:39.921041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 21902
W20260924 19:35:39.924973 21887 connection.cc:535] client connection to 127.0.98.62:40033 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40033): Connection reset by peer (error 104)
2026-09-24T19:35:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:39 dist-test-slave-m8jb krb5kdc[22001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:39 dist-test-slave-m8jb krb5kdc[22001](info): set up 2 sockets
Sep 24 19:35:39 dist-test-slave-m8jb krb5kdc[22001](info): commencing operation
krb5kdc: starting...
W20260924 19:35:40.868489 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:40 dist-test-slave-m8jb krb5kdc[22001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:40.987471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:41.067404 22014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:41.067613 22014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:41.067646 22014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:41.068809 22014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:41.068849 22014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:41.068867 22014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:41.068882 22014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:41.068898 22014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:41.070358 22014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41863
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:41.070596 22014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:41.070796 22014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:41.075649 22019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:41.075832 22014 server_base.cc:1034] running on GCE node
W20260924 19:35:41.075656 22022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:41.075660 22020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:41.076149 22014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:41.076606 22014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:41.077864 22014 hybrid_clock.cc:648] HybridClock initialized: now 1790278541077838 us; error 28 us; skew 500 ppm
Sep 24 19:35:41 dist-test-slave-m8jb krb5kdc[22001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:41.080314 22014 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:41.080796 22014 webserver.cc:460] Webserver started at http://127.0.98.62:38781/ using document root <none> and password file <none>
I20260924 19:35:41.080989 22014 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:41.081028 22014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:41.081113 22014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:41.082121 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "d398d7fad3354a649e0f8b9423c8a61f"
format_stamp: "Formatted at 2026-09-24 19:35:41 on dist-test-slave-m8jb"
server_key: "16b9c3e7ba96c3c85db2373b8f1de499"
server_key_iv: "ed68b6738f39e0dcd17cc65894da8dc3"
server_key_version: "encryptionkey@0"
I20260924 19:35:41.082419 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "d398d7fad3354a649e0f8b9423c8a61f"
format_stamp: "Formatted at 2026-09-24 19:35:41 on dist-test-slave-m8jb"
server_key: "16b9c3e7ba96c3c85db2373b8f1de499"
server_key_iv: "ed68b6738f39e0dcd17cc65894da8dc3"
server_key_version: "encryptionkey@0"
I20260924 19:35:41.083576 22014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:41.084215 22029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:41.084376 22014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:41.084440 22014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "d398d7fad3354a649e0f8b9423c8a61f"
format_stamp: "Formatted at 2026-09-24 19:35:41 on dist-test-slave-m8jb"
server_key: "16b9c3e7ba96c3c85db2373b8f1de499"
server_key_iv: "ed68b6738f39e0dcd17cc65894da8dc3"
server_key_version: "encryptionkey@0"
I20260924 19:35:41.084506 22014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:41.094610 22014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:41.096449 22014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:41.096568 22014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:41.100188 22014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41863
I20260924 19:35:41.100227 22081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41863 every 8 connection(s)
I20260924 19:35:41.100533 22014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:41.101369 22082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:41.102164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22014
I20260924 19:35:41.102267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:41.102427 392 external_mini_cluster.cc:1428] Setting key 3c93e9cd90bce9e277981d11a537ceb3
I20260924 19:35:41.103812 22082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f: Bootstrap starting.
I20260924 19:35:41.104367 22082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:41.104622 22082 log.cc:824] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:41.105316 22082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f: No bootstrap required, opened a new log
I20260924 19:35:41.107155 22082 raft_consensus.cc:348] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d398d7fad3354a649e0f8b9423c8a61f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41863 } }
I20260924 19:35:41.107297 22082 raft_consensus.cc:374] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:41.107331 22082 raft_consensus.cc:729] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d398d7fad3354a649e0f8b9423c8a61f, State: Initialized, Role: FOLLOWER
I20260924 19:35:41.107451 22082 consensus_queue.cc:260] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d398d7fad3354a649e0f8b9423c8a61f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41863 } }
I20260924 19:35:41.107524 22082 raft_consensus.cc:388] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:41.107553 22082 raft_consensus.cc:482] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:41.107616 22082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:41.107865 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:41.102903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56429 (local address 127.0.98.62:41863)
0924 19:35:41.103076 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:41.103083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:41.103107 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:35:41.103652 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:41.103658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:41.103706 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:41.103847 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:41.104040 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:41.105164 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:41.105974 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:41.106384 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:41.106477 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:41.107173 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:41.107177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:41.107177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:41.107358 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:41.107398 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:41.107413 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:41.107490 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:41.107565 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 19:35:41.107675 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260924 19:35:41.108237 22082 raft_consensus.cc:504] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d398d7fad3354a649e0f8b9423c8a61f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41863 } }
I20260924 19:35:41.108347 22082 leader_election.cc:302] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d398d7fad3354a649e0f8b9423c8a61f; no voters:
I20260924 19:35:41.108515 22082 leader_election.cc:288] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:41.108552 22087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:41.108743 22087 raft_consensus.cc:686] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [term 1 LEADER]: Becoming Leader. State: Replica: d398d7fad3354a649e0f8b9423c8a61f, State: Running, Role: LEADER
I20260924 19:35:41.108774 22082 sys_catalog.cc:564] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:41.108831 22087 consensus_queue.cc:237] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d398d7fad3354a649e0f8b9423c8a61f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41863 } }
I20260924 19:35:41.108873 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:41.109129 22087 sys_catalog.cc:455] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d398d7fad3354a649e0f8b9423c8a61f. Latest consensus state: current_term: 1 leader_uuid: "d398d7fad3354a649e0f8b9423c8a61f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d398d7fad3354a649e0f8b9423c8a61f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41863 } } }
I20260924 19:35:41.109184 22088 sys_catalog.cc:455] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d398d7fad3354a649e0f8b9423c8a61f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d398d7fad3354a649e0f8b9423c8a61f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41863 } } }
I20260924 19:35:41.109189 22087 sys_catalog.cc:458] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:41.109238 22088 sys_catalog.cc:458] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:41.109858 22094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:41.110307 22094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:41.113595 22085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:41.110376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58411 (local address 127.0.98.62:41863)
0924 19:35:41.110409 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:41.110411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:41.110547 (+ 136us) server_negotiation.cc:407] Connection header received
0924 19:35:41.110593 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:41.110593 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:41.110603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:41.110656 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:41.110785 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:41.111527 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:41.112211 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:41.112499 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:41.112570 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:41.113383 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:41.113384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:41.113385 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:41.113389 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:41.113402 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:41.113405 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:41.113455 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:41.113516 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:35:41.113524 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:41.113880 22094 catalog_manager.cc:1347] Generated new cluster ID: 07a7262b70e8448c8db2019af4851a99
I20260924 19:35:41.113919 22094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:41.130601 22094 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:41.131158 22094 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:41.134150 22094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d398d7fad3354a649e0f8b9423c8a61f: Generated new TSK 0
I20260924 19:35:41.134335 22094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:41.138192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22014
W20260924 19:35:41.141840 21999 connection.cc:535] client connection to 127.0.98.62:41863 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41863): Connection reset by peer (error 104)
2026-09-24T19:35:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:41 dist-test-slave-m8jb krb5kdc[22113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:41 dist-test-slave-m8jb krb5kdc[22113](info): set up 2 sockets
Sep 24 19:35:41 dist-test-slave-m8jb krb5kdc[22113](info): commencing operation
krb5kdc: starting...
W20260924 19:35:42.092435 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:42 dist-test-slave-m8jb krb5kdc[22113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:42.203531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37087
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:42.282516 22126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:42.282738 22126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:42.282771 22126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:42.283982 22126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:42.284034 22126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:42.284051 22126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:42.284067 22126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:42.284082 22126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:42.285703 22126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:42.285979 22126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:42.286202 22126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:42.290962 22134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:42.290963 22131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:42.291133 22132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:42.291239 22126 server_base.cc:1034] running on GCE node
I20260924 19:35:42.291488 22126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:42.291785 22126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:42.292933 22126 hybrid_clock.cc:648] HybridClock initialized: now 1790278542292916 us; error 30 us; skew 500 ppm
Sep 24 19:35:42 dist-test-slave-m8jb krb5kdc[22113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:42.295331 22126 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:42.295781 22126 webserver.cc:460] Webserver started at http://127.0.98.62:36671/ using document root <none> and password file <none>
I20260924 19:35:42.295998 22126 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:42.296046 22126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:42.296180 22126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:42.297289 22126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "3d204bcefe4049169e22d6e4c3c2b6e9"
format_stamp: "Formatted at 2026-09-24 19:35:42 on dist-test-slave-m8jb"
server_key: "246f26880ed80f00151fbe32eeebd568"
server_key_iv: "e3ad436283f98aa2938d953b4a2cad52"
server_key_version: "encryptionkey@0"
I20260924 19:35:42.297613 22126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d204bcefe4049169e22d6e4c3c2b6e9"
format_stamp: "Formatted at 2026-09-24 19:35:42 on dist-test-slave-m8jb"
server_key: "246f26880ed80f00151fbe32eeebd568"
server_key_iv: "e3ad436283f98aa2938d953b4a2cad52"
server_key_version: "encryptionkey@0"
I20260924 19:35:42.298885 22126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:42.299605 22141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:42.299757 22126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:42.299839 22126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "3d204bcefe4049169e22d6e4c3c2b6e9"
format_stamp: "Formatted at 2026-09-24 19:35:42 on dist-test-slave-m8jb"
server_key: "246f26880ed80f00151fbe32eeebd568"
server_key_iv: "e3ad436283f98aa2938d953b4a2cad52"
server_key_version: "encryptionkey@0"
I20260924 19:35:42.299898 22126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:42.303207 22126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:42.305173 22126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:42.305301 22126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:42.309180 22126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37087
I20260924 19:35:42.309335 22193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37087 every 8 connection(s)
I20260924 19:35:42.309564 22126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:42.310582 22194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:42.312858 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9: Bootstrap starting.
I20260924 19:35:42.313429 22194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:42.313683 22194 log.cc:824] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:42.314450 22194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9: No bootstrap required, opened a new log
I20260924 19:35:42.316021 22194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d204bcefe4049169e22d6e4c3c2b6e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } }
I20260924 19:35:42.316169 22194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:42.316193 22194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d204bcefe4049169e22d6e4c3c2b6e9, State: Initialized, Role: FOLLOWER
I20260924 19:35:42.316327 22194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d204bcefe4049169e22d6e4c3c2b6e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } }
I20260924 19:35:42.316390 22194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:42.316409 22194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:42.316435 22194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:42.316998 22194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d204bcefe4049169e22d6e4c3c2b6e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } }
I20260924 19:35:42.317080 22194 leader_election.cc:302] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d204bcefe4049169e22d6e4c3c2b6e9; no voters:
I20260924 19:35:42.317239 22194 leader_election.cc:288] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:42.317319 22197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:42.317469 22194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:42.317482 22197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d204bcefe4049169e22d6e4c3c2b6e9, State: Running, Role: LEADER
I20260924 19:35:42.317682 22197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d204bcefe4049169e22d6e4c3c2b6e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } }
I20260924 19:35:42.317945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22126
I20260924 19:35:42.318059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:42.318295 392 external_mini_cluster.cc:1428] Setting key 0e450ca224f2252a3f359418c4c1ff42
I20260924 19:35:42.318307 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d204bcefe4049169e22d6e4c3c2b6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d204bcefe4049169e22d6e4c3c2b6e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } } }
I20260924 19:35:42.318322 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d204bcefe4049169e22d6e4c3c2b6e9. Latest consensus state: current_term: 1 leader_uuid: "3d204bcefe4049169e22d6e4c3c2b6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d204bcefe4049169e22d6e4c3c2b6e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37087 } } }
I20260924 19:35:42.318393 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:42.318405 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9 [sys.catalog]: This master's current role is: LEADER
W20260924 19:35:42.319250 22212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:42.319315 22212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:42.319358 22207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:42.319505 22207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:42.320626 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:42.318921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54061 (local address 127.0.98.62:37087)
0924 19:35:42.319059 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:42.319068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:42.319103 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:35:42.319326 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:42.319333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:42.319368 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:42.319513 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:42.320193 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:42.320196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:42.320196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:42.320201 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:42.320244 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:42.320258 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:42.320316 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:42.320413 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 19:35:42.320451 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":49,"threads_started":1}
I20260924 19:35:42.321152 22207 catalog_manager.cc:1347] Generated new cluster ID: ce74931cb06e4c2d8125d857fb2dce4a
I20260924 19:35:42.321208 22207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:42.321519 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:42.323953 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:42.322897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54035 (local address 127.0.98.62:37087)
0924 19:35:42.322938 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:42.322940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:42.323065 (+ 125us) server_negotiation.cc:407] Connection header received
0924 19:35:42.323094 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:42.323094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:42.323111 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:42.323177 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:42.323749 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:42.323750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:42.323751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:42.323754 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:42.323771 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:42.323776 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:42.323833 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:42.323871 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 19:35:42.323883 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:35:42.338955 22207 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:42.339139 22207 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:42.343627 22207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d204bcefe4049169e22d6e4c3c2b6e9: Generated new TSK 0
I20260924 19:35:42.343842 22207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:42.351294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22126
2026-09-24T19:35:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:42 dist-test-slave-m8jb krb5kdc[22225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:42 dist-test-slave-m8jb krb5kdc[22225](info): set up 2 sockets
Sep 24 19:35:42 dist-test-slave-m8jb krb5kdc[22225](info): commencing operation
krb5kdc: starting...
W20260924 19:35:43.300431 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:43 dist-test-slave-m8jb krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:43.418579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35907
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:43.496228 22238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:43.496440 22238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:43.496470 22238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:43.497650 22238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:43.497699 22238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:43.497718 22238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:43.497754 22238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:43.497771 22238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:43.499217 22238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:43.499451 22238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:43.499650 22238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:43.504379 22243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:43.504354 22246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:43.504415 22238 server_base.cc:1034] running on GCE node
W20260924 19:35:43.504362 22244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:43.504789 22238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:43.505069 22238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:43.506238 22238 hybrid_clock.cc:648] HybridClock initialized: now 1790278543506219 us; error 27 us; skew 500 ppm
Sep 24 19:35:43 dist-test-slave-m8jb krb5kdc[22225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:43.508594 22238 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:43.509054 22238 webserver.cc:460] Webserver started at http://127.0.98.62:42461/ using document root <none> and password file <none>
I20260924 19:35:43.509259 22238 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:43.509302 22238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:43.509418 22238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:43.510430 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e"
format_stamp: "Formatted at 2026-09-24 19:35:43 on dist-test-slave-m8jb"
server_key: "7823895fd7843ecd3b4c65c0ccaa9204"
server_key_iv: "5548ed19fc1637b2433ed14ec273d7c5"
server_key_version: "encryptionkey@0"
I20260924 19:35:43.510747 22238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e"
format_stamp: "Formatted at 2026-09-24 19:35:43 on dist-test-slave-m8jb"
server_key: "7823895fd7843ecd3b4c65c0ccaa9204"
server_key_iv: "5548ed19fc1637b2433ed14ec273d7c5"
server_key_version: "encryptionkey@0"
I20260924 19:35:43.511968 22238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:35:43.512733 22253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:43.512890 22238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:43.512975 22238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e"
format_stamp: "Formatted at 2026-09-24 19:35:43 on dist-test-slave-m8jb"
server_key: "7823895fd7843ecd3b4c65c0ccaa9204"
server_key_iv: "5548ed19fc1637b2433ed14ec273d7c5"
server_key_version: "encryptionkey@0"
I20260924 19:35:43.513031 22238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:43.535893 22238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:43.538075 22238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:43.538192 22238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:43.541816 22238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35907
I20260924 19:35:43.541857 22305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35907 every 8 connection(s)
I20260924 19:35:43.542160 22238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:43.543016 22306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:43.543112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22238
I20260924 19:35:43.543246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:43.543445 392 external_mini_cluster.cc:1428] Setting key 5209a375fdae14e711664feae680b82e
I20260924 19:35:43.546204 22306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e: Bootstrap starting.
I20260924 19:35:43.546868 22306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:43.547153 22306 log.cc:824] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:43.547830 22306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e: No bootstrap required, opened a new log
Sep 24 19:35:43 dist-test-slave-m8jb krb5kdc[22225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:43.549958 22306 raft_consensus.cc:348] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35907 } }
I20260924 19:35:43.550128 22306 raft_consensus.cc:374] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:43.550168 22306 raft_consensus.cc:729] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eea2af52dc3449bbbde6f7d29ad8eb0e, State: Initialized, Role: FOLLOWER
I20260924 19:35:43.550297 22306 consensus_queue.cc:260] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35907 } }
I20260924 19:35:43.550369 22306 raft_consensus.cc:388] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:43.550390 22306 raft_consensus.cc:482] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:43.550437 22306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:43.551106 22306 raft_consensus.cc:504] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35907 } }
I20260924 19:35:43.551208 22306 leader_election.cc:302] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eea2af52dc3449bbbde6f7d29ad8eb0e; no voters:
I20260924 19:35:43.551395 22306 leader_election.cc:288] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:43.551424 22311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:43.551589 22311 raft_consensus.cc:686] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [term 1 LEADER]: Becoming Leader. State: Replica: eea2af52dc3449bbbde6f7d29ad8eb0e, State: Running, Role: LEADER
I20260924 19:35:43.551724 22306 sys_catalog.cc:564] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:43.551757 22311 consensus_queue.cc:237] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35907 } }
I20260924 19:35:43.552081 22311 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader eea2af52dc3449bbbde6f7d29ad8eb0e. Latest consensus state: current_term: 1 leader_uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35907 } } }
I20260924 19:35:43.552276 22311 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:43.552218 22312 sys_catalog.cc:455] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eea2af52dc3449bbbde6f7d29ad8eb0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35907 } } }
I20260924 19:35:43.552495 22312 sys_catalog.cc:458] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:43.552888 22320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:43.553373 22320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:43.554101 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:43.543954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33101 (local address 127.0.98.62:35907)
0924 19:35:43.544120 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:43.544128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:43.544152 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:35:43.544579 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:43.544586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:43.544680 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:43.544811 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:43.545522 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:43.546436 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:43.547436 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:43.547914 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:43.548023 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:43.550526 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:43.550530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:43.550530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:43.550541 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:43.552356 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:43.552792 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:43.552796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:43.552796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:43.552910 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:43.553209 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:43.553210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:43.553211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:43.553471 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:43.553584 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:43.553810 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:35:43.553942 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260924 19:35:43.555171 22320 catalog_manager.cc:1347] Generated new cluster ID: d018d425fb0f482e9d3d2210d509694c
I20260924 19:35:43.555194 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:43.555222 22320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:43.561748 22309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:43.556423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36593 (local address 127.0.98.62:35907)
0924 19:35:43.556456 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:43.556458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:43.556512 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:35:43.556567 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:43.556567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:43.556576 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:43.556619 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:43.557094 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:43.557568 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:43.558232 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:43.558460 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:43.558512 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:43.559535 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:43.559536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:43.559536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:43.559542 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:43.560790 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:43.561076 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:43.561077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:43.561077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:43.561127 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:43.561283 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:43.561284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:43.561285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:43.561387 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:43.561480 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:43.561575 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 19:35:43.561651 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:35:43.570214 22320 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:43.570819 22320 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:43.574095 22320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eea2af52dc3449bbbde6f7d29ad8eb0e: Generated new TSK 0
I20260924 19:35:43.574282 22320 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:43.579878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22238
2026-09-24T19:35:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1228 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:43 dist-test-slave-m8jb krb5kdc[22337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:43 dist-test-slave-m8jb krb5kdc[22337](info): set up 2 sockets
Sep 24 19:35:43 dist-test-slave-m8jb krb5kdc[22337](info): commencing operation
krb5kdc: starting...
W20260924 19:35:44.544430 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:44 dist-test-slave-m8jb krb5kdc[22337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:44.657259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:44.735668 22350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:44.735893 22350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:44.735916 22350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:44.737131 22350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:44.737180 22350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:44.737196 22350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:44.737208 22350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:44.737218 22350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:44.738693 22350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:44.738895 22350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:44.739074 22350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:44.743742 22356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:44.743757 22358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:44.743746 22355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:44.743816 22350 server_base.cc:1034] running on GCE node
I20260924 19:35:44.744223 22350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:44.744509 22350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:44.745654 22350 hybrid_clock.cc:648] HybridClock initialized: now 1790278544745638 us; error 32 us; skew 500 ppm
Sep 24 19:35:44 dist-test-slave-m8jb krb5kdc[22337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:44.748067 22350 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:44.748500 22350 webserver.cc:460] Webserver started at http://127.0.98.62:45775/ using document root <none> and password file <none>
I20260924 19:35:44.748740 22350 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:44.748790 22350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:44.748907 22350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:44.749935 22350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "5a9951df32bd470280e8fa9b7c52b649"
format_stamp: "Formatted at 2026-09-24 19:35:44 on dist-test-slave-m8jb"
server_key: "bd9dd728f927404020511719bc2b910e"
server_key_iv: "ed87413671d13ee9124eeae35dd44a30"
server_key_version: "encryptionkey@0"
I20260924 19:35:44.750263 22350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a9951df32bd470280e8fa9b7c52b649"
format_stamp: "Formatted at 2026-09-24 19:35:44 on dist-test-slave-m8jb"
server_key: "bd9dd728f927404020511719bc2b910e"
server_key_iv: "ed87413671d13ee9124eeae35dd44a30"
server_key_version: "encryptionkey@0"
I20260924 19:35:44.751472 22350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:44.752134 22365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:44.752272 22350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:44.752345 22350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "5a9951df32bd470280e8fa9b7c52b649"
format_stamp: "Formatted at 2026-09-24 19:35:44 on dist-test-slave-m8jb"
server_key: "bd9dd728f927404020511719bc2b910e"
server_key_iv: "ed87413671d13ee9124eeae35dd44a30"
server_key_version: "encryptionkey@0"
I20260924 19:35:44.752409 22350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:44.770649 22350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:44.772315 22350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:44.772446 22350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:44.776152 22350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32827
I20260924 19:35:44.776201 22417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32827 every 8 connection(s)
I20260924 19:35:44.776506 22350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:44.777316 22418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:44.779482 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649: Bootstrap starting.
I20260924 19:35:44.780007 22418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:44.780251 22418 log.cc:824] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:44.780875 22418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649: No bootstrap required, opened a new log
I20260924 19:35:44.781924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22350
I20260924 19:35:44.782047 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:44.782248 392 external_mini_cluster.cc:1428] Setting key 97b7fd02d30d6a6a0a7b3d339601bb24
I20260924 19:35:44.782696 22418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9951df32bd470280e8fa9b7c52b649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32827 } }
I20260924 19:35:44.782907 22418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:44.782954 22418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9951df32bd470280e8fa9b7c52b649, State: Initialized, Role: FOLLOWER
I20260924 19:35:44.783070 22418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9951df32bd470280e8fa9b7c52b649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32827 } }
I20260924 19:35:44.783143 22418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:44.783174 22418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:44.783218 22418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:44.784183 22418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9951df32bd470280e8fa9b7c52b649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32827 } }
I20260924 19:35:44.784287 22418 leader_election.cc:302] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a9951df32bd470280e8fa9b7c52b649; no voters:
I20260924 19:35:44.784426 22418 leader_election.cc:288] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:44.784464 22423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:44.784649 22423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [term 1 LEADER]: Becoming Leader. State: Replica: 5a9951df32bd470280e8fa9b7c52b649, State: Running, Role: LEADER
I20260924 19:35:44.784782 22423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9951df32bd470280e8fa9b7c52b649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32827 } }
I20260924 19:35:44.784682 22418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:44.785193 22425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a9951df32bd470280e8fa9b7c52b649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9951df32bd470280e8fa9b7c52b649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32827 } } }
I20260924 19:35:44.785399 22425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:44.785341 22424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a9951df32bd470280e8fa9b7c52b649. Latest consensus state: current_term: 1 leader_uuid: "5a9951df32bd470280e8fa9b7c52b649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a9951df32bd470280e8fa9b7c52b649" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32827 } } }
I20260924 19:35:44.785725 22424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:44.785831 22427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:44.786298 22427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:44.787725 22427 catalog_manager.cc:1347] Generated new cluster ID: bc12b331004446c392191723093bfd44
I20260924 19:35:44.787812 22427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:44.788040 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:44.782744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44053 (local address 127.0.98.62:32827)
0924 19:35:44.782907 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:44.782917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:44.782947 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:35:44.783060 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:44.783068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:44.783130 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:44.783254 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:44.783555 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:44.784925 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:44.786028 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:44.786607 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:44.786703 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:44.787417 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:44.787421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:44.787421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:44.787600 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:44.787638 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:44.787654 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:44.787712 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:44.787770 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 19:35:44.787827 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260924 19:35:44.788834 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:44.792420 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:44.790055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58907 (local address 127.0.98.62:32827)
0924 19:35:44.790094 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:44.790096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:44.790195 (+ 99us) server_negotiation.cc:407] Connection header received
0924 19:35:44.790220 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:44.790221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:44.790231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:44.790275 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:44.790375 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:44.790802 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:44.791432 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:44.791626 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:44.791677 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:44.792233 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:44.792234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:44.792234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:44.792238 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:44.792250 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:44.792253 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:44.792295 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:44.792335 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 19:35:44.792355 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:35:44.830516 22427 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:44.831269 22427 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:44.839321 22427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a9951df32bd470280e8fa9b7c52b649: Generated new TSK 0
I20260924 19:35:44.839537 22427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:44.858309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22350
2026-09-24T19:35:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:44 dist-test-slave-m8jb krb5kdc[22449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:44 dist-test-slave-m8jb krb5kdc[22449](info): set up 2 sockets
Sep 24 19:35:44 dist-test-slave-m8jb krb5kdc[22449](info): commencing operation
krb5kdc: starting...
W20260924 19:35:45.812601 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:45 dist-test-slave-m8jb krb5kdc[22449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:45.929366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36967
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:46.008426 22462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:46.008678 22462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:46.008713 22462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:46.009886 22462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:46.009928 22462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:46.009941 22462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:46.009953 22462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:46.009963 22462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:46.011399 22462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:46.011590 22462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:46.011760 22462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:46.016569 22470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:46.016660 22462 server_base.cc:1034] running on GCE node
W20260924 19:35:46.016778 22467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:46.016776 22468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:46.017053 22462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:46.017382 22462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:46.018555 22462 hybrid_clock.cc:648] HybridClock initialized: now 1790278546018531 us; error 39 us; skew 500 ppm
Sep 24 19:35:46 dist-test-slave-m8jb krb5kdc[22449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:46.021170 22462 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:46.021628 22462 webserver.cc:460] Webserver started at http://127.0.98.62:36067/ using document root <none> and password file <none>
I20260924 19:35:46.021829 22462 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:46.021875 22462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:46.021997 22462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:46.023028 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "165d2adbc5a149338ab83fbad6b9dad7"
format_stamp: "Formatted at 2026-09-24 19:35:46 on dist-test-slave-m8jb"
server_key: "d5f3e289f29d8103987e1a793b9c4175"
server_key_iv: "173d175e3f2f85191e82cca0f92556ff"
server_key_version: "encryptionkey@0"
I20260924 19:35:46.023356 22462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "165d2adbc5a149338ab83fbad6b9dad7"
format_stamp: "Formatted at 2026-09-24 19:35:46 on dist-test-slave-m8jb"
server_key: "d5f3e289f29d8103987e1a793b9c4175"
server_key_iv: "173d175e3f2f85191e82cca0f92556ff"
server_key_version: "encryptionkey@0"
I20260924 19:35:46.024588 22462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:46.025331 22477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:46.025486 22462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:46.025568 22462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "165d2adbc5a149338ab83fbad6b9dad7"
format_stamp: "Formatted at 2026-09-24 19:35:46 on dist-test-slave-m8jb"
server_key: "d5f3e289f29d8103987e1a793b9c4175"
server_key_iv: "173d175e3f2f85191e82cca0f92556ff"
server_key_version: "encryptionkey@0"
I20260924 19:35:46.025633 22462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:46.028861 22462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:46.030635 22462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:46.030761 22462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:46.034437 22462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36967
I20260924 19:35:46.034473 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36967 every 8 connection(s)
I20260924 19:35:46.034772 22462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:46.035734 22530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:46.038022 22530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7: Bootstrap starting.
I20260924 19:35:46.038568 22530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:46.038815 22530 log.cc:824] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:46.039520 22530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7: No bootstrap required, opened a new log
I20260924 19:35:46.041164 22530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d2adbc5a149338ab83fbad6b9dad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } }
I20260924 19:35:46.041301 22530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:35:46.041322 22530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 165d2adbc5a149338ab83fbad6b9dad7, State: Initialized, Role: FOLLOWER
I20260924 19:35:46.041453 22530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d2adbc5a149338ab83fbad6b9dad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } }
I20260924 19:35:46.041512 22530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:35:46.041533 22530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:35:46.041579 22530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:46.042083 22530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d2adbc5a149338ab83fbad6b9dad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } }
I20260924 19:35:46.042166 22530 leader_election.cc:302] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 165d2adbc5a149338ab83fbad6b9dad7; no voters:
I20260924 19:35:46.042310 22530 leader_election.cc:288] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:35:46.042352 22533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:46.042505 22533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [term 1 LEADER]: Becoming Leader. State: Replica: 165d2adbc5a149338ab83fbad6b9dad7, State: Running, Role: LEADER
I20260924 19:35:46.042547 22530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:46.042591 22533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d2adbc5a149338ab83fbad6b9dad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } }
I20260924 19:35:46.042934 22535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 165d2adbc5a149338ab83fbad6b9dad7. Latest consensus state: current_term: 1 leader_uuid: "165d2adbc5a149338ab83fbad6b9dad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d2adbc5a149338ab83fbad6b9dad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } } }
I20260924 19:35:46.043063 22534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "165d2adbc5a149338ab83fbad6b9dad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "165d2adbc5a149338ab83fbad6b9dad7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36967 } } }
I20260924 19:35:46.043100 22535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:46.043151 22534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:46.043531 22542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:46.043779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22462
I20260924 19:35:46.043881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:46.044059 22542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:46.044065 392 external_mini_cluster.cc:1428] Setting key ffd9c8a3d8b7ab29b254305311b66b5f
I20260924 19:35:46.045630 22542 catalog_manager.cc:1347] Generated new cluster ID: dcefac4d8ff74f369d462890d16d389d
I20260924 19:35:46.045686 22542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:46.046581 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:46.044502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58147 (local address 127.0.98.62:36967)
0924 19:35:46.045072 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:46.045081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:46.045110 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:35:46.045196 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:46.045202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:46.045226 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:46.045360 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:46.046107 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:46.046111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:46.046111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:46.046115 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:46.046153 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:46.046166 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:46.046221 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:46.046294 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:35:46.046411 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":66,"threads_started":1}
I20260924 19:35:46.047360 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:46.049898 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:46.048770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35739 (local address 127.0.98.62:36967)
0924 19:35:46.048798 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:46.048800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:46.048981 (+ 181us) server_negotiation.cc:407] Connection header received
0924 19:35:46.049061 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:46.049062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:46.049076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:46.049102 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:46.049705 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:46.049706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:46.049706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:35:46.049710 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:46.049722 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:35:46.049726 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:46.049750 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:46.049830 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 19:35:46.049842 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 19:35:46.066068 22542 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:46.066267 22542 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:46.077255 22542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 165d2adbc5a149338ab83fbad6b9dad7: Generated new TSK 0
I20260924 19:35:46.077463 22542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:35:46.089676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22462
2026-09-24T19:35:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1232 ms)
[----------] 18 tests from AuthTokenIssuingTest (22035 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:46 dist-test-slave-m8jb krb5kdc[22561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:46 dist-test-slave-m8jb krb5kdc[22561](info): set up 2 sockets
Sep 24 19:35:46 dist-test-slave-m8jb krb5kdc[22561](info): commencing operation
krb5kdc: starting...
W20260924 19:35:47.032486 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:47.148104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42961
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42961,127.0.98.61:43685,127.0.98.60:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:47.227021 22574 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:35:47.227253 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:47.227278 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:47.227304 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:47.228483 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:47.228528 22574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:47.228545 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:47.228562 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:47.228576 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:47.230162 22574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42961,127.0.98.61:43685,127.0.98.60:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:47.230415 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:47.230629 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:47.235348 22579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:47.235505 22582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:47.235351 22580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:47.235600 22574 server_base.cc:1034] running on GCE node
I20260924 19:35:47.235896 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:47.236189 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:47.237322 22574 hybrid_clock.cc:648] HybridClock initialized: now 1790278547237314 us; error 30 us; skew 500 ppm
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:47.239877 22574 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:47.240329 22574 webserver.cc:460] Webserver started at http://127.0.98.62:34369/ using document root <none> and password file <none>
I20260924 19:35:47.240543 22574 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:47.240588 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:47.240717 22574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:47.241789 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "7f98e5685a394b318fe242c749625fcc"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "b42625c38505b9df57eeba450a248542"
server_key_iv: "312d86bb4db2742b3fdedbd411844e2e"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.242128 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f98e5685a394b318fe242c749625fcc"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "b42625c38505b9df57eeba450a248542"
server_key_iv: "312d86bb4db2742b3fdedbd411844e2e"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.243500 22574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 19:35:47.244266 22589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:47.244431 22574 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:35:47.244526 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "7f98e5685a394b318fe242c749625fcc"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "b42625c38505b9df57eeba450a248542"
server_key_iv: "312d86bb4db2742b3fdedbd411844e2e"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.244594 22574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:47.258669 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:47.260504 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:47.260696 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:47.264418 22574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42961
I20260924 19:35:47.264467 22641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42961 every 8 connection(s)
I20260924 19:35:47.264829 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:47.265864 22642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:47.267473 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:47.272706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22574
I20260924 19:35:47.272809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:47.273001 392 external_mini_cluster.cc:1428] Setting key 9e0c0fe9af2f93f57dc4906f200eaf68
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:47.277180 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.268275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.0.98.62:42961)
0924 19:35:47.268415 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.268419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.268450 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:35:47.269072 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.269074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.269090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.269164 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.269693 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.270500 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.271341 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.271661 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.271754 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.273771 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.273778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.273778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.273785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.275498 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.275978 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.275981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.275981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.276069 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.276320 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.276321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.276321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.276531 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.276609 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.276834 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.276966 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260924 19:35:47.277184 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.268157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42961 (local address 127.0.0.1:40660)
0924 19:35:47.268329 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.268374 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.268921 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.269178 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.269181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.269451 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.269605 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.269609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.270588 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.270590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.271267 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.271269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.271752 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.271753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.271818 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.272298 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.272307 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.273692 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.275599 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.275607 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.275609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.275878 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.276135 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.276136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.276136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.276242 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.276599 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.276607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.276711 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.276814 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.277033 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":54,"thread_start_us":72,"threads_started":1}
I20260924 19:35:47.278051 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:47.278394 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.278265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60060)
0924 19:35:47.278311 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.278368 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:35:47.278581 22593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
I20260924 19:35:47.279003 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.278900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60064)
0924 19:35:47.278949 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.278981 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:35:47.279145 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43685: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 19:35:47.297223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43685
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42961,127.0.98.61:43685,127.0.98.60:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:35:47.328420 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } attempt: 1
I20260924 19:35:47.328989 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.328833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60076)
0924 19:35:47.328896 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.328949 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:35:47.329355 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.329269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60080)
0924 19:35:47.329306 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.329333 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:35:47.329463 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43685: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
W20260924 19:35:47.376606 22649 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:35:47.376863 22649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:47.376881 22649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:47.376900 22649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:47.378095 22649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:47.378140 22649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:47.378160 22649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:47.378177 22649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:47.378193 22649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:47.379720 22649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42961,127.0.98.61:43685,127.0.98.60:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:47.379952 22649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:47.380167 22649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:47.384922 22655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:47.384938 22657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:47.384940 22654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:47.385020 22649 server_base.cc:1034] running on GCE node
I20260924 19:35:47.385368 22649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:47.385663 22649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:47.386818 22649 hybrid_clock.cc:648] HybridClock initialized: now 1790278547386805 us; error 31 us; skew 500 ppm
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:47.389175 22649 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 19:35:47.389598 22649 webserver.cc:460] Webserver started at http://127.0.98.61:33521/ using document root <none> and password file <none>
I20260924 19:35:47.389812 22649 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:47.389858 22649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:47.389981 22649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:47.390995 22649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/data/instance:
uuid: "79b23165e5eb43c589232f8395f83c2b"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "68a6d50ce119adf319ae89077a0c4d59"
server_key_iv: "702d21506a24e39fcea2d33bc97bf9ba"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.391315 22649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/wal/instance:
uuid: "79b23165e5eb43c589232f8395f83c2b"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "68a6d50ce119adf319ae89077a0c4d59"
server_key_iv: "702d21506a24e39fcea2d33bc97bf9ba"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.392524 22649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:35:47.393280 22664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:47.393457 22649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:35:47.393534 22649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/wal
uuid: "79b23165e5eb43c589232f8395f83c2b"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "68a6d50ce119adf319ae89077a0c4d59"
server_key_iv: "702d21506a24e39fcea2d33bc97bf9ba"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.393599 22649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:47.397619 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } attempt: 2
I20260924 19:35:47.398046 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.397911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60084)
0924 19:35:47.397969 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.398010 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 19:35:47.398412 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.398311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60094)
0924 19:35:47.398353 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.398388 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:35:47.398497 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43685: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43685: connect: Connection refused (error 111)
I20260924 19:35:47.406646 22649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:47.408402 22649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:47.408527 22649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:47.412196 22649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43685
I20260924 19:35:47.412250 22716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43685 every 8 connection(s)
I20260924 19:35:47.412569 22649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:35:47.413508 22717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:47.415047 22717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:47.421816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22649
I20260924 19:35:47.421941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-1/wal/instance
I20260924 19:35:47.422147 392 external_mini_cluster.cc:1428] Setting key 428cff26cb3387d93384a32d50266773
I20260924 19:35:47.423566 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.415723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.0.98.62:42961)
0924 19:35:47.415778 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.415780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.415973 (+ 193us) server_negotiation.cc:407] Connection header received
0924 19:35:47.416464 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.416465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.416477 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.416535 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.417186 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.417688 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.418614 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.418814 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.418866 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.420889 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.420890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.420890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.420896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.422153 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.422537 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.422538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.422539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.422626 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.422892 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.422893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.422893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.423024 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.423084 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.423354 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.423474 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:35:47.423709 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.415690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42961 (local address 127.0.0.1:40664)
0924 19:35:47.415852 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.415893 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.416373 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.416601 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.416604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.416890 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.417112 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.417116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.417751 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.417752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.418542 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.418544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.418884 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.418885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.418999 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.419451 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.419458 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.420798 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.422227 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.422236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.422238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.422466 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.422698 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.422699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.422699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.422825 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.423080 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.423085 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.423214 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.423321 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.423488 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260924 19:35:47.424252 22717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 19:35:47.431049 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.424515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60102)
0924 19:35:47.424687 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.424698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.424738 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.424921 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.424921 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.425065 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.425112 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.425113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.425999 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.426000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.426495 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.426496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.426920 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.426921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.426982 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.427354 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.427359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.428538 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.430178 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.430178 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.430179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.430329 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.430515 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.430516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.430516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.430589 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.430810 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.430810 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.430846 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.430891 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.430980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260924 19:35:47.431181 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.424564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.0.98.61:43685)
0924 19:35:47.424816 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.424820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.424833 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:35:47.424846 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.424847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.424856 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.424908 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.425173 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.425925 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.426558 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.426859 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.426916 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.428649 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.428659 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.428660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.428667 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.430111 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.430391 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.430394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.430394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.430443 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.430634 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.430635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.430635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.430766 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.430821 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.430916 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.431051 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 19:35:47.431733 22717 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:47.432073 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.431973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42384)
0924 19:35:47.432015 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.432057 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 19:35:47.432195 22667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
I20260924 19:35:47.432567 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.432485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42392)
0924 19:35:47.432524 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.432547 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 19:35:47.432720 22717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 19:35:47.445299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35987
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42961,127.0.98.61:43685,127.0.98.60:35987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:35:47.482007 22717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } attempt: 1
I20260924 19:35:47.482555 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.482394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42398)
0924 19:35:47.482476 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.482524 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 19:35:47.482928 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.482839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42410)
0924 19:35:47.482885 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.482908 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 19:35:47.483037 22717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
I20260924 19:35:47.489629 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } attempt: 3
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:35:47.495875 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.489970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60110)
0924 19:35:47.490026 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.490037 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.490084 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.490206 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.490206 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.490354 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.490416 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.490418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.491112 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.491112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.491650 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.491652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.492000 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.492001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.492074 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.492524 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.492529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.493672 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.494994 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.494995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.494995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.495149 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.495304 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.495305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.495306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.495361 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.495571 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.495572 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.495618 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.495677 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.495773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 19:35:47.495882 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.490034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.0.98.61:43685)
0924 19:35:47.490095 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.490097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.490110 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:35:47.490139 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.490140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.490151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.490196 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.490497 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.491023 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.491723 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.491933 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.491994 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.493749 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.493750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.493750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.493758 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.494921 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.495204 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.495205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.495205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.495253 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.495409 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.495410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.495410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.495518 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.495574 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.495705 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.495798 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 19:35:47.496479 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:47.496834 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.496739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42412)
0924 19:35:47.496780 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.496806 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:35:47.497162 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.497091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42420)
0924 19:35:47.497128 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.497148 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 19:35:47.497239 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
W20260924 19:35:47.524670 22724 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:35:47.524885 22724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:47.524901 22724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:47.524920 22724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:47.526046 22724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:47.526096 22724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:47.526109 22724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:47.526120 22724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:47.526131 22724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:47.527590 22724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42961,127.0.98.61:43685,127.0.98.60:35987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:47.527817 22724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:47.528033 22724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:47.528399 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } attempt: 1
I20260924 19:35:47.528882 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.528759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42422)
0924 19:35:47.528814 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.528856 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:35:47.529241 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.529158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42424)
0924 19:35:47.529193 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.529223 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 19:35:47.529341 22642 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
W20260924 19:35:47.532806 22729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:47.532905 22724 server_base.cc:1034] running on GCE node
W20260924 19:35:47.533049 22730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:47.533057 22732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:47.533309 22724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:47.533555 22724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:47.534693 22724 hybrid_clock.cc:648] HybridClock initialized: now 1790278547534679 us; error 30 us; skew 500 ppm
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:47.537111 22724 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 19:35:47.537513 22724 webserver.cc:460] Webserver started at http://127.0.98.60:40319/ using document root <none> and password file <none>
I20260924 19:35:47.537719 22724 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:47.537760 22724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:47.537875 22724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:47.538877 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/data/instance:
uuid: "6c56f3c613ff40bdb6214073c8011ee9"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "2b3ee8e5deabb0995a60c07d4c1a3039"
server_key_iv: "940ebfde681858c2918360c279aa197a"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.539198 22724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/wal/instance:
uuid: "6c56f3c613ff40bdb6214073c8011ee9"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "2b3ee8e5deabb0995a60c07d4c1a3039"
server_key_iv: "940ebfde681858c2918360c279aa197a"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.540431 22724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 19:35:47.541204 22739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:47.541363 22724 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 19:35:47.541442 22724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/wal
uuid: "6c56f3c613ff40bdb6214073c8011ee9"
format_stamp: "Formatted at 2026-09-24 19:35:47 on dist-test-slave-m8jb"
server_key: "2b3ee8e5deabb0995a60c07d4c1a3039"
server_key_iv: "940ebfde681858c2918360c279aa197a"
server_key_version: "encryptionkey@0"
I20260924 19:35:47.541507 22724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:47.551190 22717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } attempt: 2
I20260924 19:35:47.551656 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.551533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42432)
0924 19:35:47.551588 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.551624 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 19:35:47.552029 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.551942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42444)
0924 19:35:47.551980 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.552007 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:35:47.552114 22717 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35987: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35987: connect: Connection refused (error 111)
I20260924 19:35:47.561124 22724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:47.563158 22724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:47.563287 22724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:47.567062 22724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35987
I20260924 19:35:47.567090 22791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35987 every 8 connection(s)
I20260924 19:35:47.567400 22724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:35:47.568326 22792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:47.569823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22724
I20260924 19:35:47.569911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790278290121024-392-0/minicluster-data/master-2/wal/instance
I20260924 19:35:47.569873 22792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:47.570087 392 external_mini_cluster.cc:1428] Setting key 0114c2cff4819ab3704aea5766301a13
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:35:47.578938 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.570625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40678 (local address 127.0.98.62:42961)
0924 19:35:47.570676 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.570678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.570865 (+ 187us) server_negotiation.cc:407] Connection header received
0924 19:35:47.571387 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.571389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.571400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.571454 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.571967 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.572466 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.573483 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.573769 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.573817 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.576086 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.576087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.576088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.576096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.577502 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.577980 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.577982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.577982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.578028 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.578280 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.578280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.578281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.578412 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.578468 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.578769 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.578846 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 19:35:47.579099 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.570590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42961 (local address 127.0.0.1:40678)
0924 19:35:47.570744 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.570781 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.571252 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.571484 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.571486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.571722 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.571879 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.571883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.572541 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.572542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.573402 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.573405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.573866 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.573867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.574027 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.574584 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.574593 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.575985 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.577593 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.577609 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.577611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.577918 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.578095 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.578096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.578096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.578223 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.578474 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.578481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.578627 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.578743 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.578881 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":26,"thread_start_us":86,"threads_started":1}
I20260924 19:35:47.579593 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.570580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.0.98.60:35987)
0924 19:35:47.570744 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.570753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.570781 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:35:47.571238 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.571247 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.571272 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.571366 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.571977 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.572914 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.573742 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.574206 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.574275 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.576473 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.576480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.576481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.576487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.578134 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.578489 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.578493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.578493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.578546 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.578741 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.578742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.578742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.578897 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.579356 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.579403 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.579476 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260924 19:35:47.579658 22792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:47.586153 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.579890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43685 (local address 127.0.0.1:60112)
0924 19:35:47.579935 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.579945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.579984 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.580090 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.580091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.580231 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.580278 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.580280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.581005 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.581006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.581601 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.581603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.582059 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.582059 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.582101 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.582423 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.582428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.583730 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.585277 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.585278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.585279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.585471 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.585644 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.585645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.585645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.585703 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.585904 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.585905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.585948 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.586010 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.586087 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:35:47.586211 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.579950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.0.98.61:43685)
0924 19:35:47.579990 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.579992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.580002 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:35:47.580027 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.580028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.580039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.580087 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.580339 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.580934 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.581677 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.581993 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.582055 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.583804 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.583805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.583805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.583811 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.585200 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.585536 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.585537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.585537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.585589 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.585757 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.585757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.585758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.585854 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.585898 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.586055 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.586126 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:47.586495 22792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:47.586938 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.580303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.0.98.62:42961)
0924 19:35:47.580338 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.580339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.580346 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:47.580386 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.580387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.580395 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.580445 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.580872 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.581352 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.582166 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.582375 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.582423 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.584525 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.584526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.584527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.584533 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.585894 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.586255 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.586257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.586257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.586311 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.586487 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.586488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.586488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.586610 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.586759 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.586793 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.586855 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:35:47.592947 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.586719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42464)
0924 19:35:47.586774 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.586782 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.586820 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.586908 (+ 88us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.586908 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.587045 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.587084 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.587085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.587760 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.587762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.588377 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.588379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.588833 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.588833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.588884 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.589188 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.589192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.590444 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.591930 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.591932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.591932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.592124 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.592306 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.592307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.592307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.592363 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.592579 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.592580 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.592626 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.592743 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.592831 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 19:35:47.592993 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.586745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.0.98.60:35987)
0924 19:35:47.586777 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.586779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.586808 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:35:47.586857 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.586858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.586865 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.586907 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.587147 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.587694 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.588449 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.588777 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.588826 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.590513 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.590514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.590514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.590521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.591859 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.592194 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.592195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.592195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.592251 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.592415 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.592416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.592416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.592509 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.592555 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.592775 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.592893 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":15}
I20260924 19:35:47.594398 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.587493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.0.98.61:43685)
0924 19:35:47.587529 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.587531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.587540 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:47.587569 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.587570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.587578 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.587617 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.588131 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.588666 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.589486 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.589694 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.589760 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.591934 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.591936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.591936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.591942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.593463 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.593802 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.593803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.593803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.593845 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.594003 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.594003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.594003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.594103 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.594207 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.594289 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.594340 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:47.594630 22792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9: Bootstrap starting.
I20260924 19:35:47.595534 22792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:47.595836 22792 log.cc:824] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:47.595944 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:47.596690 22792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9: No bootstrap required, opened a new log
I20260924 19:35:47.597121 22616 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7f98e5685a394b318fe242c749625fcc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40694
I20260924 19:35:47.598270 22792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } }
I20260924 19:35:47.598390 22792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c56f3c613ff40bdb6214073c8011ee9, State: Initialized, Role: FOLLOWER
I20260924 19:35:47.598466 22792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } }
I20260924 19:35:47.598753 22792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:47.598729 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.598838 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:35:47.600323 22809 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:47.600412 22809 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:47.600594 22724 master_runner.cc:186] Error getting master registration for 127.0.98.62:42961: OK, instance_id { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" instance_seqno: 1790278547263399 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:35:47.602552 22724 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:35:47.604487 22642 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } attempt: 2
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:35:47.612069 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.604755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42470)
0924 19:35:47.604798 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.604809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.604847 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.604976 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.604977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.605146 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.605192 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.605194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.606050 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.606050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.606852 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.606854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.607310 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.607311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.607367 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.607797 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.607802 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.609246 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.611007 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.611008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.611008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.611201 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.611400 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.611401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.611401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.611461 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.611746 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.611746 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.611810 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.611871 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.611976 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 19:35:47.612231 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.604810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42470 (local address 127.0.98.60:35987)
0924 19:35:47.604848 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.604849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.604858 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:47.604897 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.604897 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.604907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.604967 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.605258 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.605972 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.606941 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.607254 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.607311 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.609333 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.609334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.609335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.609341 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.610918 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.611281 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.611282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.611282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.611337 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.611508 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.611509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.611509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.611686 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.611744 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.611898 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.612014 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:35:47.613739 22642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc: Bootstrap starting.
I20260924 19:35:47.614485 22642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:47.614765 22642 log.cc:824] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:47.615567 22642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc: No bootstrap required, opened a new log
I20260924 19:35:47.616910 22642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } }
I20260924 19:35:47.617048 22642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f98e5685a394b318fe242c749625fcc, State: Initialized, Role: FOLLOWER
I20260924 19:35:47.617127 22642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } }
I20260924 19:35:47.617379 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.617441 22642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:47.617456 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:47.617574 22616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:35:47.617658 22616 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:47.618435 22616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } }
I20260924 19:35:47.619040 22766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f98e5685a394b318fe242c749625fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6c56f3c613ff40bdb6214073c8011ee9"
I20260924 19:35:47.619179 22766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:47.620131 22766 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f98e5685a394b318fe242c749625fcc in term 1.
W20260924 19:35:47.620201 22823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:47.620242 22823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:47.620362 22592 leader_election.cc:302] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c56f3c613ff40bdb6214073c8011ee9, 7f98e5685a394b318fe242c749625fcc; no voters:
I20260924 19:35:47.620463 22812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:47.620587 22812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [term 1 LEADER]: Becoming Leader. State: Replica: 7f98e5685a394b318fe242c749625fcc, State: Running, Role: LEADER
I20260924 19:35:47.620684 22812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } }
I20260924 19:35:47.620174 22574 master_runner.cc:186] Error getting master registration for 127.0.98.61:43685: OK, instance_id { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" instance_seqno: 1790278547411186 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:35:47.622380 22574 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:35:47.618875 22616 leader_election.cc:288] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [CANDIDATE]: Term 1 election: Requested vote from peers 79b23165e5eb43c589232f8395f83c2b (127.0.98.61:43685), 6c56f3c613ff40bdb6214073c8011ee9 (127.0.98.60:35987)
I20260924 19:35:47.621234 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f98e5685a394b318fe242c749625fcc. Latest consensus state: current_term: 1 leader_uuid: "7f98e5685a394b318fe242c749625fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.625033 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:47.635459 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.624942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.0.98.60:35987)
0924 19:35:47.624987 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.624989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.625242 (+ 253us) server_negotiation.cc:407] Connection header received
0924 19:35:47.626098 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.626100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.626111 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.626138 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.628541 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.629249 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.631079 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.631378 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.631415 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.632774 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.632776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.632776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.632783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.634285 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.634664 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.634665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.634666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.634719 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.634893 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.634894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.634895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.635020 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.635088 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.635280 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.635356 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:35:47.635900 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.623589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.0.98.62:42961)
0924 19:35:47.623660 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.623662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.625077 (+ 1415us) server_negotiation.cc:407] Connection header received
0924 19:35:47.625193 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.625207 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.625217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.625247 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.626376 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.627114 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.628658 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.628966 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.629006 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.631262 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.631263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.631264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.631284 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.633215 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.634369 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.634370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.634371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.634472 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.634951 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.634952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.634952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.635096 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.635163 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.635725 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.635805 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:35:47.636790 22834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:47.637070 22834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:47.640671 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.623611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.0.98.61:43685)
0924 19:35:47.623694 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.623696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.625026 (+ 1330us) server_negotiation.cc:407] Connection header received
0924 19:35:47.625337 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.625339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.625349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.625376 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.626795 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.627531 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.630250 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.630589 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.630630 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.637050 (+ 6420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.637052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.637053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.637061 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.638695 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.639219 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.639220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.639220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.639278 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.639532 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.639533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.639533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.639665 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.640159 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.640474 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.640555 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:35:47.643218 22717 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } attempt: 3
Sep 24 19:35:47 dist-test-slave-m8jb krb5kdc[22561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:35:47.651187 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.643468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35987 (local address 127.0.0.1:42496)
0924 19:35:47.643565 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:47.643575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:47.643609 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:47.643798 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:47.643799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:47.643953 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:47.643996 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.643997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.645054 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.645055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.645895 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:47.645897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.646366 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.646368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:47.646426 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.646835 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:47.646840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:47.648308 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:47.650193 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.650194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.650194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.650399 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.650583 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:47.650583 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:47.650584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:47.650642 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:47.650883 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:47.650884 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:47.650929 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:35:47.650991 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:35:47.651086 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 19:35:47.651547 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:47.643595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.98.60:35987)
0924 19:35:47.643636 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.643650 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.643658 (+ 8us) server_negotiation.cc:407] Connection header received
0924 19:35:47.643680 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.643680 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.643689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.643743 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.644063 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.644916 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.645985 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:47.646299 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:47.646361 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:47.648398 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:47.648400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:47.648400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:47.648407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:47.650117 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.650470 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.650471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.650471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.650529 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:47.650690 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:47.650691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:47.650691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:47.650825 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:47.650885 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:47.651373 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 19:35:47.651445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:35:47.653213 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b: Bootstrap starting.
I20260924 19:35:47.653982 22717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:47.654282 22717 log.cc:824] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:47.655228 22717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b: No bootstrap required, opened a new log
I20260924 19:35:47.657626 22766 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [term 1 FOLLOWER]: Refusing update from remote peer 7f98e5685a394b318fe242c749625fcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:47.657577 22717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } }
I20260924 19:35:47.657994 22825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:47.658034 22717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b23165e5eb43c589232f8395f83c2b, State: Initialized, Role: FOLLOWER
I20260924 19:35:47.659971 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f98e5685a394b318fe242c749625fcc. Latest consensus state: current_term: 1 leader_uuid: "7f98e5685a394b318fe242c749625fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.660064 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:47.660131 22717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } }
I20260924 19:35:47.660480 22717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:47.660530 22691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f98e5685a394b318fe242c749625fcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79b23165e5eb43c589232f8395f83c2b"
I20260924 19:35:47.660667 22691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:47.661525 22691 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f98e5685a394b318fe242c749625fcc in term 1.
I20260924 19:35:47.661710 22690 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [term 1 FOLLOWER]: Refusing update from remote peer 7f98e5685a394b318fe242c749625fcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:47.660477 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.662174 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:47.662540 22825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:35:47.662679 22850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:47.662726 22850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:47.663735 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f98e5685a394b318fe242c749625fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.663820 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:47.664201 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f98e5685a394b318fe242c749625fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.664276 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:47.665886 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f98e5685a394b318fe242c749625fcc. Latest consensus state: current_term: 1 leader_uuid: "7f98e5685a394b318fe242c749625fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.665961 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:47.666066 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f98e5685a394b318fe242c749625fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.666158 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:47.666662 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f98e5685a394b318fe242c749625fcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f98e5685a394b318fe242c749625fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42961 } } peers { permanent_uuid: "79b23165e5eb43c589232f8395f83c2b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43685 } } peers { permanent_uuid: "6c56f3c613ff40bdb6214073c8011ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35987 } } }
I20260924 19:35:47.666731 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:47.666846 22837 mvcc.cc:205] Tried to move back new op lower bound from 7332980931202883584 to 7332980931054997504. Current Snapshot: MvccSnapshot[applied={T|T < 7332980931202883584 or (T in {7332980931202883584})}]
I20260924 19:35:47.669041 22854 mvcc.cc:205] Tried to move back new op lower bound from 7332980931202883584 to 7332980931054997504. Current Snapshot: MvccSnapshot[applied={T|T < 7332980931202883584}]
I20260924 19:35:47.669111 22834 catalog_manager.cc:1347] Generated new cluster ID: 8bf093bfb2bb4c78b7efb4e14c0f1932
I20260924 19:35:47.669155 22834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:47.686620 22834 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:47.687346 22834 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:47.691340 22834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f98e5685a394b318fe242c749625fcc: Generated new TSK 0
I20260924 19:35:47.691509 22834 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:35:47.714468 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.712898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.0.98.61:43685)
0924 19:35:47.712953 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.712955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.713825 (+ 870us) server_negotiation.cc:407] Connection header received
0924 19:35:47.713937 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.713938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.713948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.713978 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.714394 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60140: BlockingRecv error: recv got EOF from 127.0.0.1:60140 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 19:35:47.715170 22795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.713276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.0.98.60:35987)
0924 19:35:47.713313 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.713314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.714691 (+ 1377us) server_negotiation.cc:407] Connection header received
0924 19:35:47.714781 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.714782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.714793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.714821 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.715112 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42508: BlockingRecv error: recv got EOF from 127.0.0.1:42508 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:35:47.715826 22645 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.713593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.0.98.62:42961)
0924 19:35:47.713637 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.713638 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.715366 (+ 1728us) server_negotiation.cc:407] Connection header received
0924 19:35:47.715450 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.715451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.715461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.715489 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.715754 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40716: BlockingRecv error: recv got EOF from 127.0.0.1:40716 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:35:47.722041 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.721314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60152 (local address 127.0.98.61:43685)
0924 19:35:47.721350 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.721352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.721525 (+ 173us) server_negotiation.cc:407] Connection header received
0924 19:35:47.721623 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.721623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.721642 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.721670 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.721992 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60152: BlockingRecv error: recv got EOF from 127.0.0.1:60152 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:35:47.725023 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.724229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.0.98.61:43685)
0924 19:35:47.724260 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.724262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.724500 (+ 238us) server_negotiation.cc:407] Connection header received
0924 19:35:47.724588 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.724589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.724598 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.724625 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.724975 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60164: BlockingRecv error: recv got EOF from 127.0.0.1:60164 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 19:35:47.728842 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.728027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.0.98.61:43685)
0924 19:35:47.728059 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.728060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.728285 (+ 225us) server_negotiation.cc:407] Connection header received
0924 19:35:47.728374 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.728375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.728384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.728410 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.728795 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60174: BlockingRecv error: recv got EOF from 127.0.0.1:60174 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 19:35:47.734609 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.733849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.0.98.61:43685)
0924 19:35:47.733883 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.733885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.734128 (+ 243us) server_negotiation.cc:407] Connection header received
0924 19:35:47.734218 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.734219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.734229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.734256 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.734560 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60184: BlockingRecv error: recv got EOF from 127.0.0.1:60184 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 19:35:47.744874 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.743840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.0.98.61:43685)
0924 19:35:47.743881 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.743883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.744213 (+ 330us) server_negotiation.cc:407] Connection header received
0924 19:35:47.744315 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.744317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.744328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.744357 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.744816 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60194: BlockingRecv error: recv got EOF from 127.0.0.1:60194 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:35:47.763381 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.762429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.0.98.61:43685)
0924 19:35:47.762472 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.762474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.762779 (+ 305us) server_negotiation.cc:407] Connection header received
0924 19:35:47.762881 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.762882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.762894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.762937 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.763321 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60208: BlockingRecv error: recv got EOF from 127.0.0.1:60208 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:35:47.797452 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.796704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.0.98.61:43685)
0924 19:35:47.796751 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.796753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.796886 (+ 133us) server_negotiation.cc:407] Connection header received
0924 19:35:47.796992 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.796993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.797003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.797031 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.797392 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60212: BlockingRecv error: recv got EOF from 127.0.0.1:60212 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 19:35:47.863533 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.862891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.98.61:43685)
0924 19:35:47.862986 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.862988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.863000 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:35:47.863060 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.863062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.863085 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.863143 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.863473 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60222: BlockingRecv error: recv got EOF from 127.0.0.1:60222 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 19:35:47.993319 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:47.992689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.0.98.61:43685)
0924 19:35:47.992747 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:47.992749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:47.992781 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:35:47.992890 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:47.992891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:47.992902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:47.992968 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:47.993256 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60228: BlockingRecv error: recv got EOF from 127.0.0.1:60228 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 19:35:48.251085 22721 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:48.250490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.0.98.61:43685)
0924 19:35:48.250549 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:48.250551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:48.250563 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:35:48.250616 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:48.250617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:48.250627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:48.250682 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:48.251018 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60240: BlockingRecv error: recv got EOF from 127.0.0.1:60240 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:35:48.600906 22809 catalog_manager.cc:1259] Loaded cluster ID: 8bf093bfb2bb4c78b7efb4e14c0f1932
I20260924 19:35:48.600984 22809 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9: loading cluster ID for follower catalog manager: success
I20260924 19:35:48.601779 22809 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9: acquiring CA information for follower catalog manager: success
I20260924 19:35:48.602047 22809 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c56f3c613ff40bdb6214073c8011ee9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:35:48.663117 22850 catalog_manager.cc:1259] Loaded cluster ID: 8bf093bfb2bb4c78b7efb4e14c0f1932
I20260924 19:35:48.663195 22850 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b: loading cluster ID for follower catalog manager: success
I20260924 19:35:48.664018 22850 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b: acquiring CA information for follower catalog manager: success
I20260924 19:35:48.664310 22850 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79b23165e5eb43c589232f8395f83c2b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:35:48.766718 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:48.764189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.0.98.61:43685)
0924 19:35:48.764318 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:48.764320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:48.764357 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:35:48.764428 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:48.764428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:48.764482 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:48.764559 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:48.764719 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.765290 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.766069 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.766269 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.766327 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:48.766433 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:48.766542 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:48.766597 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:48.766612 (+ 15us) server_negotiation.cc:299] Negotiation successful
0924 19:35:48.766631 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260924 19:35:48.770641 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:48.768353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.0.98.60:35987)
0924 19:35:48.768457 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:48.768458 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:48.768465 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:48.768499 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:48.768500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:48.768539 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:48.768602 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:48.768703 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.769177 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.769968 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.770168 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.770211 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:48.770375 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:48.770467 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:48.770514 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:48.770550 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 19:35:48.770566 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":49,"threads_started":1}
I20260924 19:35:48.775177 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:48.772514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60254 (local address 127.0.98.61:43685)
0924 19:35:48.772572 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:48.772575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:48.772686 (+ 111us) server_negotiation.cc:407] Connection header received
0924 19:35:48.772746 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:48.772747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:48.772758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:48.772823 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:48.773050 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.773771 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.774463 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.774718 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.774774 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:48.774918 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:48.774949 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:48.774989 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:48.775070 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 19:35:48.775098 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:35:48.775626 22927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:48.772324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.0.98.60:35987)
0924 19:35:48.772358 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:48.772360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:48.772622 (+ 262us) server_negotiation.cc:407] Connection header received
0924 19:35:48.772672 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:48.772673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:48.772685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:48.772735 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:48.772887 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.773610 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.774602 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.774862 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.774912 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:48.775301 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:48.775336 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:48.775390 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:48.775537 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:35:48.775557 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:48.775812 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:48.772361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.98.62:42961)
0924 19:35:48.772587 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:48.772588 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:48.772595 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:48.772657 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:48.772657 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:48.772706 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:48.772772 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:48.773020 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.773525 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.774634 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:48.774889 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:48.774956 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:48.775459 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:48.775559 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:48.775618 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:48.775705 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 19:35:48.775730 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":54,"threads_started":1}
I20260924 19:35:48.777282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22574
I20260924 19:35:48.781922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22649
I20260924 19:35:48.786337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22724
2026-09-24T19:35:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2695 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:35:48 dist-test-slave-m8jb krb5kdc[22938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:35:48 dist-test-slave-m8jb krb5kdc[22938](info): set up 2 sockets
Sep 24 19:35:48 dist-test-slave-m8jb krb5kdc[22938](info): commencing operation
krb5kdc: starting...
W20260924 19:35:49.732597 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/test-user.keytab.
Sep 24 19:35:49 dist-test-slave-m8jb krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 19:35:49.844367 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46767,127.0.98.61:38009,127.0.98.60:43231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:35:49.922914 22951 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:35:49.923156 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:49.923180 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:49.923206 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:49.924369 22951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:49.924412 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:49.924428 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:49.924444 22951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:49.924459 22951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:49.925971 22951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46767,127.0.98.61:38009,127.0.98.60:43231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:49.926222 22951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:49.926425 22951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:49.931098 22951 server_base.cc:1034] running on GCE node
W20260924 19:35:49.931210 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:49.931152 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:49.931098 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:49.931524 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:49.931771 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:49.932896 22951 hybrid_clock.cc:648] HybridClock initialized: now 1790278549932881 us; error 36 us; skew 500 ppm
Sep 24 19:35:49 dist-test-slave-m8jb krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:49.943110 22951 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 19:35:49.943567 22951 webserver.cc:460] Webserver started at http://127.0.98.62:46557/ using document root <none> and password file <none>
I20260924 19:35:49.943779 22951 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:49.943821 22951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:49.943939 22951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:49.945030 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/data/instance:
uuid: "756d8500618a4b8f82bb43852551c165"
format_stamp: "Formatted at 2026-09-24 19:35:49 on dist-test-slave-m8jb"
server_key: "f03fdabdf2ac49eed75734a4ef33405d"
server_key_iv: "20ec036c0700405a745f294c0449581a"
server_key_version: "encryptionkey@0"
I20260924 19:35:49.945355 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/wal/instance:
uuid: "756d8500618a4b8f82bb43852551c165"
format_stamp: "Formatted at 2026-09-24 19:35:49 on dist-test-slave-m8jb"
server_key: "f03fdabdf2ac49eed75734a4ef33405d"
server_key_iv: "20ec036c0700405a745f294c0449581a"
server_key_version: "encryptionkey@0"
I20260924 19:35:49.946543 22951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:49.947190 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:49.947363 22951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:49.947445 22951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/wal
uuid: "756d8500618a4b8f82bb43852551c165"
format_stamp: "Formatted at 2026-09-24 19:35:49 on dist-test-slave-m8jb"
server_key: "f03fdabdf2ac49eed75734a4ef33405d"
server_key_iv: "20ec036c0700405a745f294c0449581a"
server_key_version: "encryptionkey@0"
I20260924 19:35:49.947511 22951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:49.966761 22951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:49.968860 22951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:49.968974 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:49.972729 22951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46767
I20260924 19:35:49.972783 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46767 every 8 connection(s)
I20260924 19:35:49.973093 22951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/data/info.pb
I20260924 19:35:49.974020 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:49.975625 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:49.978787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 22951
I20260924 19:35:49.978899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-0/wal/instance
I20260924 19:35:49.979094 392 external_mini_cluster.cc:1428] Setting key da15f097d88663c4fd7d1e8ec5196a77
Sep 24 19:35:49 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:49.985836 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:49.976293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.0.1:42938)
0924 19:35:49.976466 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:49.976505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:49.976989 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:49.977266 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:49.977269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:49.977545 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:49.977701 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:49.977706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:49.978699 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:49.978700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:49.979757 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:49.979760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:49.980236 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:49.980237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:49.980315 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:49.980857 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:49.980865 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:49.982386 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:49.984399 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:49.984405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:49.984407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:49.984589 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:49.984855 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:49.984856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:49.984856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:49.984939 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:49.985363 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:49.985367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:49.985427 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 19:35:49.985495 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:35:49.985626 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260924 19:35:49.985843 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:49.976450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.0.98.62:46767)
0924 19:35:49.976559 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:49.976564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:49.976591 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:35:49.977113 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:49.977115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:49.977132 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:49.977207 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:49.977776 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:49.978594 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:49.979834 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:49.980141 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:49.980217 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:49.982465 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:49.982474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:49.982474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:49.982482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:49.984315 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:49.984678 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:49.984681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:49.984681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:49.984783 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:49.985004 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:49.985005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:49.985005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:49.985291 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:49.985378 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:49.985569 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 19:35:49.985702 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":82,"thread_start_us":40,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 19:35:49.986546 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:49.986907 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:49.986810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60712)
0924 19:35:49.986848 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:49.986888 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 19:35:49.987025 22967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
I20260924 19:35:49.987416 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:49.987339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60718)
0924 19:35:49.987379 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:49.987401 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 19:35:49.987602 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38009: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 19:35:50.004280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38009
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46767,127.0.98.61:38009,127.0.98.60:43231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:35:50.036787 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } attempt: 1
I20260924 19:35:50.037230 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.037112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60732)
0924 19:35:50.037164 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.037199 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:35:50.037599 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.037456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60738)
0924 19:35:50.037514 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.037566 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:35:50.037710 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38009: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
W20260924 19:35:50.082285 23026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:35:50.082496 23026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:50.082512 23026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:50.082531 23026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:50.083740 23026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:50.083782 23026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:50.083796 23026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:50.083806 23026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:50.083818 23026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:50.085327 23026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46767,127.0.98.61:38009,127.0.98.60:43231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:50.085554 23026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:50.085749 23026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:35:50.090256 23031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:50.090252 23032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:50.090423 23034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:50.090587 23026 server_base.cc:1034] running on GCE node
I20260924 19:35:50.090819 23026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:50.091133 23026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:50.092280 23026 hybrid_clock.cc:648] HybridClock initialized: now 1790278550092259 us; error 35 us; skew 500 ppm
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:50.094628 23026 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 19:35:50.095113 23026 webserver.cc:460] Webserver started at http://127.0.98.61:34167/ using document root <none> and password file <none>
I20260924 19:35:50.095309 23026 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:50.095350 23026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:50.095469 23026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:50.096462 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/data/instance:
uuid: "bc24797f0d76442797763cd4145b94b7"
format_stamp: "Formatted at 2026-09-24 19:35:50 on dist-test-slave-m8jb"
server_key: "eb2945efe74cc30bb3920275ae5a88b7"
server_key_iv: "66fb87f54af878dc8a20e481c60d820b"
server_key_version: "encryptionkey@0"
I20260924 19:35:50.096803 23026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/wal/instance:
uuid: "bc24797f0d76442797763cd4145b94b7"
format_stamp: "Formatted at 2026-09-24 19:35:50 on dist-test-slave-m8jb"
server_key: "eb2945efe74cc30bb3920275ae5a88b7"
server_key_iv: "66fb87f54af878dc8a20e481c60d820b"
server_key_version: "encryptionkey@0"
I20260924 19:35:50.097955 23026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 19:35:50.098572 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:50.098692 23026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 19:35:50.098765 23026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/wal
uuid: "bc24797f0d76442797763cd4145b94b7"
format_stamp: "Formatted at 2026-09-24 19:35:50 on dist-test-slave-m8jb"
server_key: "eb2945efe74cc30bb3920275ae5a88b7"
server_key_iv: "66fb87f54af878dc8a20e481c60d820b"
server_key_version: "encryptionkey@0"
I20260924 19:35:50.098827 23026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:50.105842 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } attempt: 2
I20260924 19:35:50.106251 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.106134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60754)
0924 19:35:50.106185 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.106222 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 19:35:50.106577 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.106482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60766)
0924 19:35:50.106521 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.106547 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:35:50.106673 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38009: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38009: connect: Connection refused (error 111)
I20260924 19:35:50.114117 23026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:50.116132 23026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:50.116251 23026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:50.120044 23026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38009
I20260924 19:35:50.120075 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38009 every 8 connection(s)
I20260924 19:35:50.120405 23026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/data/info.pb
I20260924 19:35:50.121327 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:50.122777 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:50.128793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 23026
I20260924 19:35:50.128897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-1/wal/instance
I20260924 19:35:50.129099 392 external_mini_cluster.cc:1428] Setting key c1036fc5cd66e92199b8285f8470a29d
I20260924 19:35:50.130991 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.123408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.0.98.62:46767)
0924 19:35:50.123453 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.123455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.123639 (+ 184us) server_negotiation.cc:407] Connection header received
0924 19:35:50.124109 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.124109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.124118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.124175 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.124770 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.125246 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.126120 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.126310 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.126354 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.128242 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.128243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.128244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.128251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.129622 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.130064 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.130065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.130065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.130114 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.130376 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.130377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.130377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.130522 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.130589 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.130807 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.130892 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 19:35:50.131067 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.123379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.0.1:42940)
0924 19:35:50.123528 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.123568 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:50.124029 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:50.124237 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:50.124240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:50.124473 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:50.124702 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.124708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.125305 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.125306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.126059 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.126061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.126375 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.126376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.126462 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.126856 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:50.126864 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:50.128133 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:50.129698 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.129707 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.129709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.129995 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.130165 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.130166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.130166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.130301 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.130581 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:50.130585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:50.130698 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 19:35:50.130777 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:35:50.130902 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260924 19:35:50.131590 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 19:35:50.138610 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.131836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60770)
0924 19:35:50.131880 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.131890 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:50.131926 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:50.132126 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:50.132127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:50.132250 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:50.132294 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.132295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.133582 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.133583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.134062 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.134063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.134646 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.134647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.134697 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.135065 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:50.135070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:50.136185 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:50.137703 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.137704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.137704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.137848 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.138033 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.138034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.138034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.138101 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.138377 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:50.138378 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:50.138415 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:35:50.138468 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:35:50.138546 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 19:35:50.138708 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.131870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.0.98.61:38009)
0924 19:35:50.132016 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.132021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.132036 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:35:50.132051 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.132053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.132062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.132121 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.132374 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.133502 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.134133 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.134583 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.134652 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.136257 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.136264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.136264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.136270 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.137645 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.137905 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.137907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.137907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.137953 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.138165 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.138166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.138166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.138312 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.138372 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.138490 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.138601 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 19:35:50.139257 23094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:50.139566 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.139446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33592)
0924 19:35:50.139494 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.139543 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 19:35:50.139703 23043 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
I20260924 19:35:50.140072 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.139982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33602)
0924 19:35:50.140035 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.140056 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 19:35:50.140185 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 19:35:50.151862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLXPSqI/build/release/bin/kudu
/tmp/dist-test-taskLXPSqI/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43231
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46767,127.0.98.61:38009,127.0.98.60:43231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:35:50.189419 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } attempt: 1
I20260924 19:35:50.189963 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.189807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33618)
0924 19:35:50.189887 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.189931 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:35:50.190361 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.190271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33624)
0924 19:35:50.190316 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.190339 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 19:35:50.190440 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
I20260924 19:35:50.197811 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } attempt: 3
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:35:50.205297 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.198156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60774)
0924 19:35:50.198216 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.198229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:50.198284 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:50.198442 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:50.198443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:50.198599 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:50.198673 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.198675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.199608 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.199609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.200175 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.200177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.200698 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.200700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.200766 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.201248 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:50.201253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:50.202734 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:50.204404 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.204404 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.204405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.204566 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.204767 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.204767 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.204768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.204828 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.205046 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:50.205046 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:50.205083 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:35:50.205132 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 19:35:50.205225 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 19:35:50.205324 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.198206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.0.98.61:38009)
0924 19:35:50.198270 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.198273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.198285 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:35:50.198334 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.198335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.198349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.198409 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.198755 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.199515 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.200249 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.200583 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.200687 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.202824 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.202825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.202826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.202833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.204332 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.204624 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.204625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.204626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.204703 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.204873 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.204874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.204874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.204996 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.205054 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.205153 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.205232 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:35:50.205688 23019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:50.205994 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.205905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33626)
0924 19:35:50.205945 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.205970 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 19:35:50.206307 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.206236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33632)
0924 19:35:50.206273 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.206293 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 19:35:50.206387 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
W20260924 19:35:50.232170 23101 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:35:50.232383 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:35:50.232398 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:35:50.232417 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:35:50.233502 23101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:35:50.233536 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:35:50.233548 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:35:50.233559 23101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:35:50.233570 23101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:35:50.234963 23101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46767,127.0.98.61:38009,127.0.98.60:43231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 18:36:13 UTC on ccycloud.quasar-yyhgly.root.comops.site
build id 4670860
I20260924 19:35:50.235162 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:35:50.235330 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:35:50.237550 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } attempt: 1
I20260924 19:35:50.238029 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.237858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33646)
0924 19:35:50.237945 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.237996 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 19:35:50.238380 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.238285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33658)
0924 19:35:50.238330 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.238358 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 19:35:50.238462 23019 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
W20260924 19:35:50.239837 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:50.239831 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:35:50.239847 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:35:50.239956 23101 server_base.cc:1034] running on GCE node
I20260924 19:35:50.240258 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:35:50.240520 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:35:50.241648 23101 hybrid_clock.cc:648] HybridClock initialized: now 1790278550241638 us; error 30 us; skew 500 ppm
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:35:50.243945 23101 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 19:35:50.244354 23101 webserver.cc:460] Webserver started at http://127.0.98.60:45059/ using document root <none> and password file <none>
I20260924 19:35:50.244534 23101 fs_manager.cc:359] Metadata directory not provided
I20260924 19:35:50.244572 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:35:50.244699 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:35:50.245687 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/data/instance:
uuid: "9e415c4295e147a9b01a3bc65cf007a5"
format_stamp: "Formatted at 2026-09-24 19:35:50 on dist-test-slave-m8jb"
server_key: "6f69c55c21a4c504517f0b5b0a09485c"
server_key_iv: "36d1313d8058c387e6e56258f8dadb7d"
server_key_version: "encryptionkey@0"
I20260924 19:35:50.245987 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/wal/instance:
uuid: "9e415c4295e147a9b01a3bc65cf007a5"
format_stamp: "Formatted at 2026-09-24 19:35:50 on dist-test-slave-m8jb"
server_key: "6f69c55c21a4c504517f0b5b0a09485c"
server_key_iv: "36d1313d8058c387e6e56258f8dadb7d"
server_key_version: "encryptionkey@0"
I20260924 19:35:50.247118 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:35:50.247712 23116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:35:50.247851 23101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 19:35:50.247929 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/wal
uuid: "9e415c4295e147a9b01a3bc65cf007a5"
format_stamp: "Formatted at 2026-09-24 19:35:50 on dist-test-slave-m8jb"
server_key: "6f69c55c21a4c504517f0b5b0a09485c"
server_key_iv: "36d1313d8058c387e6e56258f8dadb7d"
server_key_version: "encryptionkey@0"
I20260924 19:35:50.247994 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:35:50.258563 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } attempt: 2
I20260924 19:35:50.258970 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.258861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33664)
0924 19:35:50.258910 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.258941 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 19:35:50.259276 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.259198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33674)
0924 19:35:50.259234 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.259257 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 19:35:50.259349 23094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43231: connect: Connection refused (error 111)
I20260924 19:35:50.263971 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:35:50.265813 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:35:50.265942 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:35:50.269549 23101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43231
I20260924 19:35:50.269589 23168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43231 every 8 connection(s)
I20260924 19:35:50.269891 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/data/info.pb
I20260924 19:35:50.270776 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:35:50.272251 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:35:50.276202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLXPSqI/build/release/bin/kudu as pid 23101
I20260924 19:35:50.276309 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLXPSqI/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790278290121024-392-0/minicluster-data/master-2/wal/instance
I20260924 19:35:50.276520 392 external_mini_cluster.cc:1428] Setting key 4543ef760b8eef2e7b55217120236276
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 19:35:50.280819 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.272895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.0.98.62:46767)
0924 19:35:50.272956 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.272959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.273124 (+ 165us) server_negotiation.cc:407] Connection header received
0924 19:35:50.273622 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.273623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.273634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.273684 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.274252 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.274712 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.275614 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.275798 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.275852 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.278101 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.278102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.278102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.278109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.279447 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.279868 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.279868 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.279869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.279919 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.280148 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.280148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.280148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.280272 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.280339 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.280647 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.280728 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:35:50.280916 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.272855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46767 (local address 127.0.0.1:42948)
0924 19:35:50.273012 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.273049 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:50.273520 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:50.273747 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:50.273751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:50.273982 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:50.274167 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.274171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.274785 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.274786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.275535 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.275537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.275881 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.275882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.275989 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.276498 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:50.276507 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:50.278026 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:50.279527 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.279532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.279534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.279800 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.279969 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.279970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.279970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.280095 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.280331 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:50.280335 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:50.280483 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 19:35:50.280590 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:35:50.280747 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260924 19:35:50.281446 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:35:50.285223 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.276876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.98.60:43231)
0924 19:35:50.277023 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.277027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.277044 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:35:50.277108 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.277110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.277127 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.277200 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.277747 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.278651 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.279653 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.280084 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.280149 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.282584 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.282591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.282592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.282598 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.284072 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.284416 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.284418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.284418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.284466 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.284678 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.284679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.284679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.284843 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.285018 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.285061 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.285129 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260924 19:35:50.287623 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.281759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.0.98.61:38009)
0924 19:35:50.281801 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.281803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.281810 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:50.281833 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.281834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.281844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.281883 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.282138 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.282597 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.283403 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.283651 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.283705 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.285485 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.285486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.285487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.285494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.286711 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.286989 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.286990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.286990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.287044 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.287193 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.287194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.287194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.287297 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.287347 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.287470 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.287535 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:35:50.287623 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.281693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38009 (local address 127.0.0.1:60780)
0924 19:35:50.281737 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.281748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:50.281787 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:50.281889 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:50.281890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:50.282029 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:50.282080 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.282082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.282645 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.282646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.283326 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.283328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.283724 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.283725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.283795 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.284157 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:50.284161 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:50.285410 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:50.286781 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.286782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.286783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.286925 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.287093 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.287094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.287094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.287145 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.287345 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:50.287346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:50.287384 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:35:50.287442 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:35:50.287534 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 19:35:50.288156 23169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:35:50.292305 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.285960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.0.98.62:46767)
0924 19:35:50.285997 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.286000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.286046 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:35:50.286102 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.286102 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.286111 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.286153 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.286549 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.287088 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.287700 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.287897 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.287946 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.290053 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.290055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.290055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.290062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.291382 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.291672 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.291673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.291673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.291722 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.291874 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.291875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.291875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.291975 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.292125 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.292159 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.292223 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:50.294147 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.288366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33684)
0924 19:35:50.288460 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.288469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:50.288523 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:50.288649 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:50.288650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:50.288779 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:50.288811 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.288813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.289468 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.289469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.289964 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.289965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.290299 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.290300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.290341 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.290558 (+ 217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:50.290563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:50.291645 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:50.293296 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.293297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.293297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.293497 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.293683 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.293684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.293684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.293738 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.293951 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:50.293952 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:50.293983 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:35:50.294030 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 19:35:50.294089 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 19:35:50.294239 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.288422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.0.98.60:43231)
0924 19:35:50.288454 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.288456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.288510 (+ 54us) server_negotiation.cc:407] Connection header received
0924 19:35:50.288573 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.288574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.288588 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.288624 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.288870 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.289397 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.290033 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.290234 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.290293 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.291743 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.291744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.291744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.291751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.293223 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.293577 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.293578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.293579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.293626 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.293791 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.293792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.293792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.293898 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.293956 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.294065 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.294154 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":17}
I20260924 19:35:50.295885 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5: Bootstrap starting.
I20260924 19:35:50.296893 23169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:50.297150 23169 log.cc:824] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 19:35:50.297823 23169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5: No bootstrap required, opened a new log
I20260924 19:35:50.299400 23169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } }
I20260924 19:35:50.299540 23169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e415c4295e147a9b01a3bc65cf007a5, State: Initialized, Role: FOLLOWER
I20260924 19:35:50.299618 23169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } }
I20260924 19:35:50.299754 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.293126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.0.98.61:38009)
0924 19:35:50.293173 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.293175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.293205 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:35:50.293249 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.293250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.293261 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.293329 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.293828 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.294436 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.295245 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.295470 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.295540 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.297683 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.297684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.297685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.297691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.298861 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.299143 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.299144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.299145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.299190 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.299340 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.299341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.299341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.299435 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.299573 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.299610 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.299672 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 19:35:50.300005 23169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:50.300058 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.300151 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:50.301118 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:35:50.301957 22993 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "756d8500618a4b8f82bb43852551c165"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42950
W20260924 19:35:50.301693 23186 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:50.303057 23186 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:50.303376 23101 master_runner.cc:186] Error getting master registration for 127.0.98.62:46767: OK, instance_id { permanent_uuid: "756d8500618a4b8f82bb43852551c165" instance_seqno: 1790278549971662 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:35:50.305631 23101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:35:50.313601 23019 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } attempt: 2
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:35:50.321122 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.313932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33694)
0924 19:35:50.314000 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.314011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:50.314050 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:50.314243 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:50.314244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:50.314430 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:50.314486 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.314489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.315404 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.315405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.316194 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.316195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.316655 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.316655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.316709 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.317176 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:50.317181 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:50.318466 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:50.320107 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.320108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.320108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.320264 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.320450 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.320451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.320451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.320509 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.320818 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:50.320819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:50.320866 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 19:35:50.320928 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 19:35:50.321017 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 19:35:50.321136 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.313989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33694 (local address 127.0.98.60:43231)
0924 19:35:50.314029 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.314031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.314043 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:35:50.314146 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.314147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.314158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.314210 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.314562 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.315321 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.316265 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.316563 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.316677 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.318553 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.318554 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.318554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.318562 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.320036 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.320323 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.320325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.320325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.320381 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.320560 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.320561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.320561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.320759 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.320828 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.320960 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.321043 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 19:35:50.322607 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165: Bootstrap starting.
I20260924 19:35:50.323199 23019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:50.323447 23019 log.cc:824] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:50.324093 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165: No bootstrap required, opened a new log
I20260924 19:35:50.325124 23019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } }
I20260924 19:35:50.325237 23019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756d8500618a4b8f82bb43852551c165, State: Initialized, Role: FOLLOWER
I20260924 19:35:50.325305 23019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } }
I20260924 19:35:50.325538 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.325587 23019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:50.325624 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:50.325729 22993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:35:50.325806 22993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:50.326567 22993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } }
W20260924 19:35:50.327476 23200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:50.327570 23200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:50.327966 23143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "756d8500618a4b8f82bb43852551c165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e415c4295e147a9b01a3bc65cf007a5"
I20260924 19:35:50.328095 23143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:50.328987 23143 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 756d8500618a4b8f82bb43852551c165 in term 1.
I20260924 19:35:50.329217 22968 leader_election.cc:302] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 756d8500618a4b8f82bb43852551c165, 9e415c4295e147a9b01a3bc65cf007a5; no voters:
I20260924 19:35:50.329352 23189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:35:50.329473 23189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [term 1 LEADER]: Becoming Leader. State: Replica: 756d8500618a4b8f82bb43852551c165, State: Running, Role: LEADER
I20260924 19:35:50.329548 23189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } }
I20260924 19:35:50.329967 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 756d8500618a4b8f82bb43852551c165. Latest consensus state: current_term: 1 leader_uuid: "756d8500618a4b8f82bb43852551c165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.330049 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:50.329560 22993 leader_election.cc:288] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [CANDIDATE]: Term 1 election: Requested vote from peers bc24797f0d76442797763cd4145b94b7 (127.0.98.61:38009), 9e415c4295e147a9b01a3bc65cf007a5 (127.0.98.60:43231)
I20260924 19:35:50.331135 23204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:35:50.331317 23204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:35:50.329530 22951 master_runner.cc:186] Error getting master registration for 127.0.98.61:38009: OK, instance_id { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" instance_seqno: 1790278550118966 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:35:50.331849 22951 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:35:50.333895 23143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [term 1 FOLLOWER]: Refusing update from remote peer 756d8500618a4b8f82bb43852551c165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:50.334539 23202 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:50.335978 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 756d8500618a4b8f82bb43852551c165. Latest consensus state: current_term: 1 leader_uuid: "756d8500618a4b8f82bb43852551c165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.336074 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:50.343943 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.331785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.0.98.60:43231)
0924 19:35:50.331823 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.331825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.332953 (+ 1128us) server_negotiation.cc:407] Connection header received
0924 19:35:50.333059 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.333060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.333071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.333101 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.336113 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.336866 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.337909 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.338221 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.338301 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.340244 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.340245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.340245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.340253 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.341972 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.342532 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.342533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.342533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.342593 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.342799 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.342800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.342800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.342928 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.343029 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.343754 (+ 725us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.343848 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 19:35:50.345831 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "756d8500618a4b8f82bb43852551c165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.345930 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: This master's current role is: LEADER
I20260924 19:35:50.346313 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "756d8500618a4b8f82bb43852551c165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.346407 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:50.346933 23210 mvcc.cc:205] Tried to move back new op lower bound from 7332980942164451328 to 7332980942150668288. Current Snapshot: MvccSnapshot[applied={T|T < 7332980942164451328}]
I20260924 19:35:50.347580 23213 mvcc.cc:205] Tried to move back new op lower bound from 7332980942164451328 to 7332980942150668288. Current Snapshot: MvccSnapshot[applied={T|T < 7332980942164451328}]
I20260924 19:35:50.348065 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.332549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.0.98.62:46767)
0924 19:35:50.332583 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.332585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.338007 (+ 5422us) server_negotiation.cc:407] Connection header received
0924 19:35:50.338095 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.338096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.338105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.338133 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.338811 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.339528 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.341559 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.341843 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.341883 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.344439 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.344441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.344442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.344449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.345969 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.346827 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.346828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.346829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.346904 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.347193 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.347194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.347194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.347332 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.347440 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.347879 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.347970 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 19:35:50.348116 23204 catalog_manager.cc:1347] Generated new cluster ID: a252646cba804d14845d9982b0ae5536
I20260924 19:35:50.348155 23204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:35:50.350689 23094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } attempt: 3
I20260924 19:35:50.353197 23098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.332878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.0.98.61:38009)
0924 19:35:50.332919 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.332921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.340763 (+ 7842us) server_negotiation.cc:407] Connection header received
0924 19:35:50.340855 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.340857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.340869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.340896 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.341461 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.342184 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.343125 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.343469 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.343530 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.349951 (+ 6421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.349953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.349953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.349962 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.351928 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.352266 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.352268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.352268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.352321 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.352465 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.352465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.352466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.352595 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.353002 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.353044 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.353114 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 19:35:50 dist-test-slave-m8jb krb5kdc[22938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790278550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 19:35:50.361120 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.350929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43231 (local address 127.0.0.1:33716)
0924 19:35:50.351007 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 19:35:50.351017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 19:35:50.351051 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:35:50.352189 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:35:50.352190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:35:50.352377 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:35:50.352424 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.352426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.353726 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.353727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.354511 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:35:50.354513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.355245 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.355246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:35:50.355310 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.355669 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:35:50.355688 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:35:50.357165 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:35:50.359759 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.359760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.359761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.359968 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.360267 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:35:50.360268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:35:50.360268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:35:50.360359 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:35:50.360823 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:35:50.360824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:35:50.360881 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 19:35:50.360927 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 19:35:50.361022 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:35:50.361929 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:50.351192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33716 (local address 127.0.98.60:43231)
0924 19:35:50.352049 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.352051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.352058 (+ 7us) server_negotiation.cc:407] Connection header received
0924 19:35:50.352091 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.352092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.352102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.352506 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.352551 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.353611 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.354652 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:50.355130 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:50.355597 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:50.357274 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:35:50.357275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:35:50.357276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:35:50.357322 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:35:50.359650 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.360059 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.360060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.360060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.360173 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:35:50.360440 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:35:50.360440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:35:50.360441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:35:50.360699 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:35:50.361550 (+ 851us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:50.361626 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 19:35:50.361756 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836}
I20260924 19:35:50.362295 23204 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:35:50.362991 23204 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:35:50.363569 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7: Bootstrap starting.
I20260924 19:35:50.364255 23094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7: Neither blocks nor log segments found. Creating new log.
I20260924 19:35:50.364539 23094 log.cc:824] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:35:50.365598 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7: No bootstrap required, opened a new log
I20260924 19:35:50.367633 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } }
I20260924 19:35:50.367761 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc24797f0d76442797763cd4145b94b7, State: Initialized, Role: FOLLOWER
I20260924 19:35:50.367834 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } }
I20260924 19:35:50.368144 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:35:50.368227 23204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165: Generated new TSK 0
I20260924 19:35:50.368392 23204 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:35:50.369966 23230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:35:50.370030 23230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:35:50.369526 23068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "756d8500618a4b8f82bb43852551c165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc24797f0d76442797763cd4145b94b7"
I20260924 19:35:50.369678 23067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:35:50.371603 23067 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [term 1 FOLLOWER]: Refusing update from remote peer 756d8500618a4b8f82bb43852551c165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:35:50.373521 23068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 756d8500618a4b8f82bb43852551c165 in term 1.
I20260924 19:35:50.374243 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.374416 23189 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:35:50.374322 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:50.377086 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 756d8500618a4b8f82bb43852551c165. Latest consensus state: current_term: 1 leader_uuid: "756d8500618a4b8f82bb43852551c165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.379261 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:35:50.378363 23233 mvcc.cc:205] Tried to move back new op lower bound from 7332980942301999104 to 7332980942150668288. Current Snapshot: MvccSnapshot[applied={T|T < 7332980942164451328}]
I20260924 19:35:50.379588 23232 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "756d8500618a4b8f82bb43852551c165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.379707 23232 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:35:50.392854 23173 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.391272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.0.98.60:43231)
0924 19:35:50.391313 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.391315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.392212 (+ 897us) server_negotiation.cc:407] Connection header received
0924 19:35:50.392311 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.392312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.392323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.392349 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.392791 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33722: BlockingRecv error: recv got EOF from 127.0.0.1:33722 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 19:35:50.393599 23022 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.391657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.0.98.62:46767)
0924 19:35:50.391695 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.391697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.393093 (+ 1396us) server_negotiation.cc:407] Connection header received
0924 19:35:50.393185 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.393186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.393197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.393227 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.393535 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42960: BlockingRecv error: recv got EOF from 127.0.0.1:42960 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:35:50.394253 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.391959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.0.98.61:38009)
0924 19:35:50.392012 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.392014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.393812 (+ 1798us) server_negotiation.cc:407] Connection header received
0924 19:35:50.393873 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.393874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.393886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.393934 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.394194 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60804: BlockingRecv error: recv got EOF from 127.0.0.1:60804 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 19:35:50.396154 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.395410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.0.98.61:38009)
0924 19:35:50.395448 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.395450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.395657 (+ 207us) server_negotiation.cc:407] Connection header received
0924 19:35:50.395758 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.395758 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.395775 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.395803 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.396110 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60812: BlockingRecv error: recv got EOF from 127.0.0.1:60812 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 19:35:50.398998 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.398343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.0.98.61:38009)
0924 19:35:50.398373 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.398375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.398563 (+ 188us) server_negotiation.cc:407] Connection header received
0924 19:35:50.398645 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.398646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.398655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.398679 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.398945 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60822: BlockingRecv error: recv got EOF from 127.0.0.1:60822 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 19:35:50.405786 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.404896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.98.61:38009)
0924 19:35:50.404935 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.404937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.405230 (+ 293us) server_negotiation.cc:407] Connection header received
0924 19:35:50.405328 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.405329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.405342 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.405372 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.405729 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60834: BlockingRecv error: recv got EOF from 127.0.0.1:60834 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 19:35:50.411877 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.411050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.0.98.61:38009)
0924 19:35:50.411089 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.411091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.411321 (+ 230us) server_negotiation.cc:407] Connection header received
0924 19:35:50.411417 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.411418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.411430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.411458 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.411822 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60844: BlockingRecv error: recv got EOF from 127.0.0.1:60844 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:35:50.422132 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.421261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.0.98.61:38009)
0924 19:35:50.421303 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.421305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.421543 (+ 238us) server_negotiation.cc:407] Connection header received
0924 19:35:50.421644 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.421645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.421656 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.421686 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.422072 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60856: BlockingRecv error: recv got EOF from 127.0.0.1:60856 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 19:35:50.440371 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.439481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.0.98.61:38009)
0924 19:35:50.439526 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.439529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.439775 (+ 246us) server_negotiation.cc:407] Connection header received
0924 19:35:50.439878 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.439879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.439892 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.439921 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.440314 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60862: BlockingRecv error: recv got EOF from 127.0.0.1:60862 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 19:35:50.477118 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.476502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.0.98.61:38009)
0924 19:35:50.476549 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.476552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.476561 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:50.476588 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.476589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.476601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.476682 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.477059 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60878: BlockingRecv error: recv got EOF from 127.0.0.1:60878 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 19:35:50.543133 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.542436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.0.98.61:38009)
0924 19:35:50.542489 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.542491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.542555 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:35:50.542634 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.542635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.542644 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.542688 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.543075 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60890: BlockingRecv error: recv got EOF from 127.0.0.1:60890 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260924 19:35:50.672840 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.672203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.0.98.61:38009)
0924 19:35:50.672260 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.672262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.672272 (+ 10us) server_negotiation.cc:407] Connection header received
0924 19:35:50.672325 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.672326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.672335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.672393 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.672711 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60906: BlockingRecv error: recv got EOF from 127.0.0.1:60906 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 19:35:50.861701 23189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "756d8500618a4b8f82bb43852551c165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756d8500618a4b8f82bb43852551c165" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46767 } } peers { permanent_uuid: "bc24797f0d76442797763cd4145b94b7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38009 } } peers { permanent_uuid: "9e415c4295e147a9b01a3bc65cf007a5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43231 } } }
I20260924 19:35:50.861835 23189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756d8500618a4b8f82bb43852551c165 [sys.catalog]: This master's current role is: LEADER
W20260924 19:35:50.930377 23098 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:35:50.929832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.0.98.61:38009)
0924 19:35:50.929895 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:50.929897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:50.929910 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:35:50.929974 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:50.929975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:50.929986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:50.930037 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:35:50.930314 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60914: BlockingRecv error: recv got EOF from 127.0.0.1:60914 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 19:35:51.303661 23186 catalog_manager.cc:1259] Loaded cluster ID: a252646cba804d14845d9982b0ae5536
I20260924 19:35:51.303735 23186 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5: loading cluster ID for follower catalog manager: success
I20260924 19:35:51.304584 23186 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5: acquiring CA information for follower catalog manager: success
I20260924 19:35:51.304895 23186 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e415c4295e147a9b01a3bc65cf007a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:35:51.370404 23230 catalog_manager.cc:1259] Loaded cluster ID: a252646cba804d14845d9982b0ae5536
I20260924 19:35:51.370482 23230 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7: loading cluster ID for follower catalog manager: success
I20260924 19:35:51.371236 23230 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7: acquiring CA information for follower catalog manager: success
I20260924 19:35:51.371505 23230 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc24797f0d76442797763cd4145b94b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:35:51.446040 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:51.443494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.0.98.61:38009)
0924 19:35:51.443607 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:51.443609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:51.443618 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:51.443695 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:51.443696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:51.443747 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:51.443852 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:51.443997 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.444496 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.445290 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.445489 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.445545 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:51.445704 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:51.445818 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:51.445884 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:51.445931 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 19:35:51.445953 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":50,"threads_started":1}
I20260924 19:35:51.449785 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:51.447653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.0.98.60:43231)
0924 19:35:51.447741 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:51.447743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:51.447752 (+ 9us) server_negotiation.cc:407] Connection header received
0924 19:35:51.447788 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:51.447788 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:51.447822 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:51.447885 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:51.447968 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.448421 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.449208 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.449388 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.449433 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:51.449549 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:51.449638 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:51.449682 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:51.449697 (+ 15us) server_negotiation.cc:299] Negotiation successful
0924 19:35:51.449712 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":41,"threads_started":1}
I20260924 19:35:51.454071 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:51.451402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.0.98.60:43231)
0924 19:35:51.451438 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:51.451440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:51.451658 (+ 218us) server_negotiation.cc:407] Connection header received
0924 19:35:51.451702 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:51.451703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:51.451712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:51.451762 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:51.451903 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.452373 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.453326 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.453596 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.453649 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:51.453807 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:51.453839 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:51.453888 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:51.453981 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 19:35:51.454001 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 19:35:51.454435 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:51.451502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.0.98.61:38009)
0924 19:35:51.451539 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:51.451540 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:51.451727 (+ 187us) server_negotiation.cc:407] Connection header received
0924 19:35:51.451785 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:51.451787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:51.451796 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:51.451845 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:51.451980 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.452710 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.453632 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.453991 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.454042 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:51.454218 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:51.454247 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:51.454271 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:51.454344 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:35:51.454365 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 19:35:51.454530 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:35:51.451437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.0.98.62:46767)
0924 19:35:51.451569 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 19:35:51.451571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 19:35:51.451601 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:35:51.451682 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:35:51.451683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:35:51.451730 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:35:51.451798 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:35:51.451940 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.452752 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.453596 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:35:51.453858 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:35:51.453913 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:35:51.454115 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:35:51.454220 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:35:51.454288 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:35:51.454426 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:35:51.454452 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260924 19:35:51.455986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 22951
I20260924 19:35:51.460207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 23026
I20260924 19:35:51.464334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLXPSqI/build/release/bin/kudu with pid 23101
2026-09-24T19:35:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2678 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5373 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (261348 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST